Jul 19 14:36:25.947202 ip-10-0-134-156 systemd[1]: Starting Kubernetes Kubelet... Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.407586 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413888 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413900 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413904 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413907 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413910 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413914 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413917 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:36:26.493617 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413920 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:36:26.489774 ip-10-0-134-156 systemd[1]: Started Kubernetes Kubelet. Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413923 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413926 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413928 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413950 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413954 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413957 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413960 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413963 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413966 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413969 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413972 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413974 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413978 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413980 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413983 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413986 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413989 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413992 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413995 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.413997 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:36:26.496256 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414000 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414004 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414009 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414012 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414015 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414018 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414020 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414023 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414026 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414029 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414031 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414033 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414036 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414039 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414042 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414044 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414048 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414051 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414053 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414055 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:36:26.510257 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414058 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414061 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414064 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414067 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414069 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414072 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414075 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414078 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414080 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414083 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414085 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414088 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414091 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414093 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414096 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414099 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414102 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414104 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414107 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414109 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:36:26.514790 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414112 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414114 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414117 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414119 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414122 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414124 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414127 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414129 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414132 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414136 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414140 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414143 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414146 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414149 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414151 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414154 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414157 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414160 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414162 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:36:26.517217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414165 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414169 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414172 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414175 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414177 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414181 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414183 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414186 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414188 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414191 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414194 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414196 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414883 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414889 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414892 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414895 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414900 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414903 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414907 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:36:26.519463 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414910 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414913 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414916 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414919 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414922 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414925 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414928 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414949 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414952 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414955 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414958 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414960 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414963 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414966 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414968 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414971 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414973 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414976 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414979 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414981 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:36:26.521579 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414984 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414986 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414990 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414992 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414995 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.414997 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415000 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415002 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415005 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415008 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415011 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415013 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415015 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415018 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415020 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415023 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415025 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415028 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415031 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415035 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:36:26.522878 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415038 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415040 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415044 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415048 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415052 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415054 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415057 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415059 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415062 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415064 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415067 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415069 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415072 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415074 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415077 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415080 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415082 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415085 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415088 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:36:26.523792 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415090 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415093 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415095 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415098 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415100 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415103 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415105 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415107 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415110 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415112 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415115 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415118 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415125 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415127 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415130 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415132 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415135 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415137 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415140 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415142 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:36:26.524750 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415145 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415147 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415151 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415155 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415157 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415161 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415164 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415166 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415169 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415172 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415174 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415177 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415179 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415538 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415543 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415546 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415548 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415551 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415554 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:36:26.525583 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415556 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415559 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415561 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415564 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415568 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415575 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415578 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415581 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415584 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415586 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415588 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415591 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415593 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415596 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415598 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415601 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415603 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415606 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415608 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415611 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:36:26.526285 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415614 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415616 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415619 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415621 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415624 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415626 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415629 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415631 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415634 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415636 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415639 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415641 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415643 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415646 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415648 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415651 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415653 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415659 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415661 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:36:26.678565 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415666 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415669 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415672 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415675 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415678 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415680 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415683 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415686 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415688 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415691 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415694 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415696 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415699 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415701 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415704 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415706 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415709 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415711 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415714 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415716 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:36:26.680335 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415719 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415721 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415723 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415726 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415728 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415730 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415734 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415737 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415740 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415744 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415748 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415752 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415755 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415757 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415760 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415763 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415765 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415768 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415770 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415773 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:36:26.681202 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415775 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415777 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415780 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415782 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415785 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415787 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415790 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415792 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415796 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415799 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415802 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415804 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415807 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.415809 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415879 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415890 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415896 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415900 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415905 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415908 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 19 14:36:26.682217 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415912 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415916 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415922 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415925 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415930 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415949 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415953 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415957 2559 flags.go:64] FLAG: --cgroup-root="" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415960 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415963 2559 flags.go:64] FLAG: --client-ca-file="" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415965 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415969 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415973 2559 flags.go:64] FLAG: --cluster-domain="" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415976 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415979 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415982 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415986 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415990 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415993 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.415996 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416000 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416003 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416006 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416009 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 19 14:36:26.683260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416022 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416025 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416029 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416032 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416035 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416038 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416041 2559 flags.go:64] FLAG: --enable-server="true" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416044 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416048 2559 flags.go:64] FLAG: --event-burst="100" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416051 2559 flags.go:64] FLAG: --event-qps="50" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416057 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416060 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416063 2559 flags.go:64] FLAG: --eviction-hard="" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416067 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416070 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416074 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416077 2559 flags.go:64] FLAG: --eviction-soft="" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416080 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416083 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416085 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416088 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416091 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416094 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416097 2559 flags.go:64] FLAG: --feature-gates="" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416100 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 19 14:36:26.684287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416104 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416107 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416110 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416113 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416115 2559 flags.go:64] FLAG: --help="false" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416118 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416121 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416124 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416127 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416131 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416134 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416137 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416139 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416142 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416145 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416147 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416150 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416156 2559 flags.go:64] FLAG: --kube-reserved="" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416158 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416161 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416164 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416168 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416171 2559 flags.go:64] FLAG: --lock-file="" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416174 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 19 14:36:26.685119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416177 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416179 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416185 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416188 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416191 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416194 2559 flags.go:64] FLAG: --logging-format="text" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416197 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416200 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416202 2559 flags.go:64] FLAG: --manifest-url="" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416205 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416210 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416213 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416216 2559 flags.go:64] FLAG: --max-pods="110" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416219 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416222 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416225 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416228 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416231 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416233 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416237 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416244 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416246 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416249 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 19 14:36:26.685961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416252 2559 flags.go:64] FLAG: --pod-cidr="" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416255 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416261 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416265 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416269 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416272 2559 flags.go:64] FLAG: --port="10250" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416275 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416278 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fde8773f983c0f8b" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416281 2559 flags.go:64] FLAG: --qos-reserved="" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416284 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416287 2559 flags.go:64] FLAG: --register-node="true" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416290 2559 flags.go:64] FLAG: --register-with-taints="" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416294 2559 flags.go:64] FLAG: --registry-burst="10" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416296 2559 flags.go:64] FLAG: --registry-qps="5" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416299 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416302 2559 flags.go:64] FLAG: --reserved-memory="" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416305 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416308 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416311 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416314 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416316 2559 flags.go:64] FLAG: --runonce="false" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416319 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416323 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416326 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416328 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 19 14:36:26.687364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416331 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416334 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416338 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416341 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416343 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416346 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416349 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416351 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416354 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416358 2559 flags.go:64] FLAG: --system-cgroups="" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416363 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416368 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416371 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416374 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416378 2559 flags.go:64] FLAG: --tls-min-version="" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416381 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416384 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416387 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416390 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416393 2559 flags.go:64] FLAG: --v="2" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416397 2559 flags.go:64] FLAG: --version="false" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416401 2559 flags.go:64] FLAG: --vmodule="" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416404 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416407 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416516 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:36:26.688087 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416520 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416523 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416526 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416529 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416531 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416534 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416537 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416539 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416542 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416545 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416547 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416550 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416552 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416555 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416557 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416560 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416564 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416568 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416571 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:36:26.810452 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416574 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416578 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416581 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416584 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416588 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416591 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416594 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416596 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416599 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416601 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416604 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416606 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416609 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416611 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416615 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416619 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416621 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416624 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416626 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:36:26.811179 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416629 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416632 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416634 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416637 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416640 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416642 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416645 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416647 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416650 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416652 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416656 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416659 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416662 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416664 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416667 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416669 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416672 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416674 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416677 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416680 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:36:26.811723 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416682 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416685 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416687 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416690 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416692 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416695 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416697 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416700 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416702 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416705 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416707 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416709 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416712 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416714 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416717 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416720 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416722 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416725 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416727 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416730 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:36:26.812318 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416732 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416734 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416738 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416742 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416744 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416747 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416749 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416752 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416755 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416757 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416760 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416767 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416770 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416774 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416778 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416780 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416783 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416786 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416789 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:36:26.812829 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.416792 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.416797 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.424074 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.424090 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.424985 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.424992 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425048 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425053 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425057 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425060 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425074 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425077 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425082 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425086 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425089 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:36:26.813393 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425093 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425096 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425099 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425104 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425107 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425110 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425113 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425116 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425119 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425123 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425126 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425128 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425131 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425133 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425136 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425139 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425142 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425145 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425148 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:36:26.813806 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425152 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425156 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425160 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425163 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425166 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425169 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425172 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425175 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425178 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425181 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425184 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425187 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425189 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425192 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425195 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425197 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425200 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425202 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425205 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:36:26.814352 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425208 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425211 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425214 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425216 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425219 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425222 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425224 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425228 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425230 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425233 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425235 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425239 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425242 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425244 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425247 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425249 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425252 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425254 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425257 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:36:26.814892 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425259 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425262 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425264 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425267 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425270 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425273 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425275 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425278 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425280 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425283 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425286 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425290 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425293 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425296 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425298 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425301 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425303 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425307 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425310 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425312 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:36:26.815446 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425315 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425317 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425320 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425322 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425325 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425328 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425330 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425333 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425335 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425338 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425340 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425343 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425345 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.425351 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:36:26.815978 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425463 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425467 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425470 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425473 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425476 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425478 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425481 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425484 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425486 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425491 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425493 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425496 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425498 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425501 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425504 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425506 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425509 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425512 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425515 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425517 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:36:26.816399 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425520 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425522 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425525 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425528 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425530 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425533 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425535 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425538 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425541 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425544 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425546 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425549 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425552 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425554 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425556 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425559 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425561 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425564 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425567 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425569 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:36:26.816925 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425572 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425576 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425580 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425583 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425585 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425588 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425590 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425593 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425596 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425598 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425601 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425604 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425608 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425611 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425614 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425617 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425619 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425622 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425625 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:36:26.817510 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425628 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425631 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425633 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425636 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425639 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425641 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425644 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425646 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425648 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425652 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425654 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425657 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425659 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425662 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425666 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425669 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425672 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425674 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425678 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:36:26.818024 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425682 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425684 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425687 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425689 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425692 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425699 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425702 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425704 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425707 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425709 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425712 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425714 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425717 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425719 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425723 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425726 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425729 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425732 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425735 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425737 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:36:26.818590 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:26.425740 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.425745 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.425850 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.428997 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.430569 2559 server.go:1019] "Starting client certificate rotation" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.430673 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.430705 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.434693 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.436963 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.450184 2559 log.go:25] "Validated CRI v1 runtime API" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.452708 2559 log.go:25] "Validated CRI v1 image API" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.454080 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 19 14:36:26.819116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.457124 2559 fs.go:135] Filesystem UUIDs: map[1def3ec7-b9c5-4980-8120-c6ef368830d7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8338ffa8-f6f8-4c4c-bc07-1ebb5a133a32:/dev/nvme0n1p4] Jul 19 14:36:26.819493 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.457140 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 19 14:36:26.819493 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.459953 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.462773 2559 manager.go:216] Machine: {Timestamp:2026-07-19 14:36:26.461260226 +0000 UTC m=+0.380414450 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106844 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aca966e07b3d5357187a16f677ee1 SystemUUID:ec2aca96-6e07-b3d5-3571-87a16f677ee1 BootID:90d09578-7201-4414-9381-7fb4b3694475 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:10:6e:ef:d2:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:10:6e:ef:d2:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:b6:30:42:3e:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.462874 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.462964 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.466383 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.466407 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.467222 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.467232 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.467241 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.468060 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.469901 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.474673 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.474812 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.474848 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.474866 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.474881 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.474893 2559 kubelet.go:398] "Adding apiserver pod source" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.474904 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 19 14:36:26.819656 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.475316 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.475336 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.477755 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.479649 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.479677 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480463 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480477 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480490 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480496 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480505 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480510 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480519 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480524 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480530 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480536 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480550 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.480555 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.481446 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.481462 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.484534 2559 server.go:1268] "Started kubelet" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.484604 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.484628 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.489140 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.492332 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.492661 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.492685 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.492686 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.494754 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.496169 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-156.ec2.internal.18c3b7a2a11da885 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-156.ec2.internal,UID:ip-10-0-134-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-156.ec2.internal,},FirstTimestamp:2026-07-19 14:36:26.484508805 +0000 UTC m=+0.403663022,LastTimestamp:2026-07-19 14:36:26.484508805 +0000 UTC m=+0.403663022,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-156.ec2.internal,}" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.497758 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.504004 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.504009 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.504057 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.504709 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.504720 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.504737 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.504842 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.504851 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.504877 2559 factory.go:55] Registering systemd factory Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.504903 2559 factory.go:223] Registration of the systemd container factory successfully Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.505183 2559 factory.go:153] Registering CRI-O factory Jul 19 14:36:26.820512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.505194 2559 factory.go:223] Registration of the crio container factory successfully Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.505243 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.505262 2559 factory.go:103] Registering Raw factory Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.505271 2559 manager.go:1196] Started watching for new ooms in manager Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.505675 2559 manager.go:349] Starting recovery of all containers Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.506061 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jmzkf" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.506250 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.506339 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.511337 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jmzkf" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.514308 2559 manager.go:354] Recovery completed Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.520720 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.523648 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.523686 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.523701 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.525608 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.525621 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.525624 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.525636 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.526406 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.526438 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.526454 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.529205 2559 policy_none.go:49] "None policy: Start" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.529222 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.529234 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.531230 2559 policy_none.go:47] "Start" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.574499 2559 manager.go:337] "Starting Device Plugin manager" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.574529 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.574542 2559 server.go:87] "Starting device plugin registration server" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.574804 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.574815 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.574897 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.574989 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.574996 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.575564 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.575620 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.602535 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.603629 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.603669 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.603694 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.603732 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.606091 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.675327 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.682292 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.682328 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.682344 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.682376 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.821771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.691981 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.692000 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-156.ec2.internal\": node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.704279 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"] Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.704336 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.705792 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.705818 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.705833 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.708054 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.708155 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.708180 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.708810 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.708837 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.708865 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.708867 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.708889 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.708905 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.711116 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.711146 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.712529 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.712552 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.712564 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.719722 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.752511 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-156.ec2.internal\" not found" node="ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.756780 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-156.ec2.internal\" not found" node="ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.806462 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/85c349124b4cc1127652c4cd0349da6a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"85c349124b4cc1127652c4cd0349da6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.806483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85c349124b4cc1127652c4cd0349da6a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"85c349124b4cc1127652c4cd0349da6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.806499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d3debe7f1b24cd0f146d976ae1a3f64-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"3d3debe7f1b24cd0f146d976ae1a3f64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.820542 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.906919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/85c349124b4cc1127652c4cd0349da6a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"85c349124b4cc1127652c4cd0349da6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.989639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.906969 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85c349124b4cc1127652c4cd0349da6a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"85c349124b4cc1127652c4cd0349da6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.990529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.906985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d3debe7f1b24cd0f146d976ae1a3f64-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"3d3debe7f1b24cd0f146d976ae1a3f64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.990529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.907025 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85c349124b4cc1127652c4cd0349da6a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"85c349124b4cc1127652c4cd0349da6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.990529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.907025 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d3debe7f1b24cd0f146d976ae1a3f64-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"3d3debe7f1b24cd0f146d976ae1a3f64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.990529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:26.907024 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/85c349124b4cc1127652c4cd0349da6a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"85c349124b4cc1127652c4cd0349da6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 19 14:36:26.990529 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:26.921015 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:27.143234 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:27.021962 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:27.143234 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:27.054120 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 19 14:36:27.143234 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:27.065877 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 19 14:36:27.143234 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:27.122248 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:27.306201 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:27.222762 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:27.484857 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:27.323241 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:27.484857 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:27.423842 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:27.484857 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:27.429983 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 19 14:36:27.484857 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:27.430099 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 14:36:29.132795 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:27.501019 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:36:29.132795 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.017644 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-18 14:31:28 +0000 UTC" deadline="2028-01-12 03:11:15.470642849 +0000 UTC" Jul 19 14:36:29.132795 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.017668 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12996h34m46.452977906s" Jul 19 14:36:29.132795 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:29.021786 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:29.132795 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:29.122602 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 19 14:36:29.264104 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.173334 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:36:29.264104 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.201434 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 19 14:36:29.264104 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.213740 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 14:36:29.264104 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.214689 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 19 14:36:29.264104 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.221260 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 14:36:29.264104 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.233072 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:36:29.355787 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:29.290600 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c349124b4cc1127652c4cd0349da6a.slice/crio-e4ab016fb548536cd5d14199780fa5217ca7f1effad780cdcc4b241c9c2aa1c4 WatchSource:0}: Error finding container e4ab016fb548536cd5d14199780fa5217ca7f1effad780cdcc4b241c9c2aa1c4: Status 404 returned error can't find the container with id e4ab016fb548536cd5d14199780fa5217ca7f1effad780cdcc4b241c9c2aa1c4 Jul 19 14:36:29.355787 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.295795 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 14:36:29.355787 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:29.300526 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3debe7f1b24cd0f146d976ae1a3f64.slice/crio-fdadd02f15d922371792d58ebdde3dfbab076212db667c1f877b5b58cc628bbd WatchSource:0}: Error finding container fdadd02f15d922371792d58ebdde3dfbab076212db667c1f877b5b58cc628bbd: Status 404 returned error can't find the container with id fdadd02f15d922371792d58ebdde3dfbab076212db667c1f877b5b58cc628bbd Jul 19 14:36:29.827334 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.753768 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.973212 2559 apiserver.go:52] "Watching apiserver" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.980853 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.983035 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-7rprf","openshift-multus/multus-additional-cni-plugins-qnjcn","openshift-network-operator/iptables-alerter-4zhwl","openshift-ovn-kubernetes/ovnkube-node-w4xw9","kube-system/global-pull-secret-syncer-ffhs6","kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mfsm5","openshift-image-registry/node-ca-lgdc4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal","openshift-multus/multus-cxvqm","openshift-multus/network-metrics-daemon-zw8ds","openshift-network-diagnostics/network-check-target-5dnnv","kube-system/konnectivity-agent-cdbwf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857"] Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.985478 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.987950 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.990114 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.992385 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.994944 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:29.995007 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.997066 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.000258 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:29.999409 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.003126 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.001369 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 19 14:36:30.003126 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.002456 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.004745 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.004400 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.008638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.007082 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.010548 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.009826 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:30.010548 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.009924 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:30.013534 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.012825 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-socket-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.013534 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.012880 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-device-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.013534 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.012910 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/300ecf05-298e-4db7-b26d-f7b7abeb1855-agent-certs\") pod \"konnectivity-agent-cdbwf\" (UID: \"300ecf05-298e-4db7-b26d-f7b7abeb1855\") " pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:30.013534 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.012941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/300ecf05-298e-4db7-b26d-f7b7abeb1855-konnectivity-ca\") pod \"konnectivity-agent-cdbwf\" (UID: \"300ecf05-298e-4db7-b26d-f7b7abeb1855\") " pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:30.013534 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.012975 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d502afb-2fd3-4687-a99c-ffb76d684ea6-host-slash\") pod \"iptables-alerter-4zhwl\" (UID: \"1d502afb-2fd3-4687-a99c-ffb76d684ea6\") " pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017440 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-node-log\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017484 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-cni-bin\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-etc-openvswitch\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017552 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f97c53-b5ae-4a60-887a-eedcc84baf37-ovn-node-metrics-cert\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/096a094e-c379-445d-8e2f-5eb81a93f05a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017595 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-var-lib-openvswitch\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.017667 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017675 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-tmp\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017766 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-os-release\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fbww\" (UniqueName: \"kubernetes.io/projected/7b223bb6-1dec-46f4-a6a8-7204b718d631-kube-api-access-8fbww\") pod \"node-ca-lgdc4\" (UID: \"7b223bb6-1dec-46f4-a6a8-7204b718d631\") " pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017836 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-kubelet\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-run-ovn\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.083237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f97c53-b5ae-4a60-887a-eedcc84baf37-ovnkube-script-lib\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017937 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-registration-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.017969 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5rft\" (UniqueName: \"kubernetes.io/projected/096a094e-c379-445d-8e2f-5eb81a93f05a-kube-api-access-v5rft\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqlx\" (UniqueName: \"kubernetes.io/projected/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-kube-api-access-fgqlx\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018036 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6796\" (UniqueName: \"kubernetes.io/projected/c4f97c53-b5ae-4a60-887a-eedcc84baf37-kube-api-access-q6796\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018059 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-sysconfig\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018088 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-sysctl-d\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-slash\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f97c53-b5ae-4a60-887a-eedcc84baf37-ovnkube-config\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/096a094e-c379-445d-8e2f-5eb81a93f05a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018282 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/522a5dd5-0b75-4721-830e-42417948a408-kubelet-config\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018323 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-run-netns\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018373 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-system-cni-dir\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018428 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/096a094e-c379-445d-8e2f-5eb81a93f05a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018461 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b223bb6-1dec-46f4-a6a8-7204b718d631-host\") pod \"node-ca-lgdc4\" (UID: \"7b223bb6-1dec-46f4-a6a8-7204b718d631\") " pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018512 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-run-openvswitch\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.085486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018547 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018596 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-run-systemd\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f97c53-b5ae-4a60-887a-eedcc84baf37-env-overrides\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018668 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-run\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018720 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-modprobe-d\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-systemd\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-host\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d502afb-2fd3-4687-a99c-ffb76d684ea6-iptables-alerter-script\") pod \"iptables-alerter-4zhwl\" (UID: \"1d502afb-2fd3-4687-a99c-ffb76d684ea6\") " pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jxh\" (UniqueName: \"kubernetes.io/projected/1d502afb-2fd3-4687-a99c-ffb76d684ea6-kube-api-access-r2jxh\") pod \"iptables-alerter-4zhwl\" (UID: \"1d502afb-2fd3-4687-a99c-ffb76d684ea6\") " pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.018993 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-etc-selinux\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-kubernetes\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019063 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-tuned\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-cni-netd\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-sys-fs\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019163 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-cnibin\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.086341 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019193 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-var-lib-kubelet\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019225 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019251 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b223bb6-1dec-46f4-a6a8-7204b718d631-serviceca\") pod \"node-ca-lgdc4\" (UID: \"7b223bb6-1dec-46f4-a6a8-7204b718d631\") " pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0307f2d-fa2f-42dd-984e-148f44510d07-hosts-file\") pod \"node-resolver-7rprf\" (UID: \"e0307f2d-fa2f-42dd-984e-148f44510d07\") " pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019347 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-systemd-units\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019381 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-lib-modules\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkp5\" (UniqueName: \"kubernetes.io/projected/e0307f2d-fa2f-42dd-984e-148f44510d07-kube-api-access-jfkp5\") pod \"node-resolver-7rprf\" (UID: \"e0307f2d-fa2f-42dd-984e-148f44510d07\") " pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019444 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-sys\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019468 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzml\" (UniqueName: \"kubernetes.io/projected/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-kube-api-access-wjzml\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-sysctl-conf\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019530 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/522a5dd5-0b75-4721-830e-42417948a408-dbus\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019559 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0307f2d-fa2f-42dd-984e-148f44510d07-tmp-dir\") pod \"node-resolver-7rprf\" (UID: \"e0307f2d-fa2f-42dd-984e-148f44510d07\") " pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.019594 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-log-socket\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.020582 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8z9bb\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.020639 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.020659 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.020948 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021023 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021097 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021195 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021329 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021385 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021474 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:36:30.089355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021496 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021515 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021659 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021681 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021729 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021736 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021841 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021662 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.021844 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rgq7n\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.022535 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.022899 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.023030 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.023189 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.023326 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6c9kd\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.023362 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bx6tl\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.023584 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.024011 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.024199 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-527sj\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.024376 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rwj4z\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.024436 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zs8z7\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.024654 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.024757 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.024886 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ktwws\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.025003 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.025148 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.025330 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ggpxf\"" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.032982 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.055408 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nnprf" Jul 19 14:36:30.090855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.062816 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nnprf" Jul 19 14:36:30.103497 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.102843 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-cni-dir\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119780 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-socket-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119805 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-device-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/300ecf05-298e-4db7-b26d-f7b7abeb1855-agent-certs\") pod \"konnectivity-agent-cdbwf\" (UID: \"300ecf05-298e-4db7-b26d-f7b7abeb1855\") " pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119852 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/300ecf05-298e-4db7-b26d-f7b7abeb1855-konnectivity-ca\") pod \"konnectivity-agent-cdbwf\" (UID: \"300ecf05-298e-4db7-b26d-f7b7abeb1855\") " pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d502afb-2fd3-4687-a99c-ffb76d684ea6-host-slash\") pod \"iptables-alerter-4zhwl\" (UID: \"1d502afb-2fd3-4687-a99c-ffb76d684ea6\") " pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-node-log\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-cni-bin\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119967 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-socket-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.119977 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-etc-openvswitch\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120016 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f97c53-b5ae-4a60-887a-eedcc84baf37-ovn-node-metrics-cert\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120028 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-etc-openvswitch\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120046 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-var-lib-kubelet\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120071 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-etc-kubernetes\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120085 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-device-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nct6\" (UniqueName: \"kubernetes.io/projected/82fc003a-678c-4e3d-90ab-0c64f693f817-kube-api-access-7nct6\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:30.182013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/096a094e-c379-445d-8e2f-5eb81a93f05a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120152 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-conf-dir\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-var-lib-openvswitch\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-tmp\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120273 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-hostroot\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-os-release\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fbww\" (UniqueName: \"kubernetes.io/projected/7b223bb6-1dec-46f4-a6a8-7204b718d631-kube-api-access-8fbww\") pod \"node-ca-lgdc4\" (UID: \"7b223bb6-1dec-46f4-a6a8-7204b718d631\") " pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-kubelet\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120367 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-run-ovn\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f97c53-b5ae-4a60-887a-eedcc84baf37-ovnkube-script-lib\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120403 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120796 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d502afb-2fd3-4687-a99c-ffb76d684ea6-host-slash\") pod \"iptables-alerter-4zhwl\" (UID: \"1d502afb-2fd3-4687-a99c-ffb76d684ea6\") " pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120964 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-node-log\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-var-lib-openvswitch\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/300ecf05-298e-4db7-b26d-f7b7abeb1855-konnectivity-ca\") pod \"konnectivity-agent-cdbwf\" (UID: \"300ecf05-298e-4db7-b26d-f7b7abeb1855\") " pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:30.183250 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-cni-bin\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121466 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/096a094e-c379-445d-8e2f-5eb81a93f05a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121510 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-os-release\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121536 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-run-ovn\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-kubelet\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.120423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-socket-dir-parent\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121579 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121598 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-registration-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121624 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5rft\" (UniqueName: \"kubernetes.io/projected/096a094e-c379-445d-8e2f-5eb81a93f05a-kube-api-access-v5rft\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqlx\" (UniqueName: \"kubernetes.io/projected/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-kube-api-access-fgqlx\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-registration-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121725 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6796\" (UniqueName: \"kubernetes.io/projected/c4f97c53-b5ae-4a60-887a-eedcc84baf37-kube-api-access-q6796\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-sysconfig\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-sysctl-d\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-slash\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f97c53-b5ae-4a60-887a-eedcc84baf37-ovnkube-config\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.184273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121856 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnp9j\" (UniqueName: \"kubernetes.io/projected/a756c7f8-fc87-4f5b-9d5c-803a4d580084-kube-api-access-rnp9j\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/096a094e-c379-445d-8e2f-5eb81a93f05a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/522a5dd5-0b75-4721-830e-42417948a408-kubelet-config\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-run-netns\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.121969 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-system-cni-dir\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/096a094e-c379-445d-8e2f-5eb81a93f05a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122014 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-sysctl-d\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122023 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b223bb6-1dec-46f4-a6a8-7204b718d631-host\") pod \"node-ca-lgdc4\" (UID: \"7b223bb6-1dec-46f4-a6a8-7204b718d631\") " pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122045 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-run-openvswitch\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f97c53-b5ae-4a60-887a-eedcc84baf37-ovnkube-script-lib\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122107 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122110 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-run-k8s-cni-cncf-io\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122150 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-slash\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122164 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-run-netns\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-run-systemd\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f97c53-b5ae-4a60-887a-eedcc84baf37-env-overrides\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.185157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a756c7f8-fc87-4f5b-9d5c-803a4d580084-cni-binary-copy\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122243 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-sysconfig\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-run\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122303 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-modprobe-d\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122327 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-systemd\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-host\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122368 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d502afb-2fd3-4687-a99c-ffb76d684ea6-iptables-alerter-script\") pod \"iptables-alerter-4zhwl\" (UID: \"1d502afb-2fd3-4687-a99c-ffb76d684ea6\") " pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jxh\" (UniqueName: \"kubernetes.io/projected/1d502afb-2fd3-4687-a99c-ffb76d684ea6-kube-api-access-r2jxh\") pod \"iptables-alerter-4zhwl\" (UID: \"1d502afb-2fd3-4687-a99c-ffb76d684ea6\") " pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-daemon-config\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-etc-selinux\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122442 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/522a5dd5-0b75-4721-830e-42417948a408-kubelet-config\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-kubernetes\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-run-netns\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122514 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-tuned\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122540 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-cni-netd\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122565 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-system-cni-dir\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.186324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-modprobe-d\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122592 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-run-multus-certs\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122616 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/096a094e-c379-445d-8e2f-5eb81a93f05a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-sys-fs\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-cnibin\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-var-lib-kubelet\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122704 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-systemd\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122756 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-host\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b223bb6-1dec-46f4-a6a8-7204b718d631-serviceca\") pod \"node-ca-lgdc4\" (UID: \"7b223bb6-1dec-46f4-a6a8-7204b718d631\") " pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0307f2d-fa2f-42dd-984e-148f44510d07-hosts-file\") pod \"node-resolver-7rprf\" (UID: \"e0307f2d-fa2f-42dd-984e-148f44510d07\") " pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-systemd-units\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122851 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-var-lib-cni-bin\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-lib-modules\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkp5\" (UniqueName: \"kubernetes.io/projected/e0307f2d-fa2f-42dd-984e-148f44510d07-kube-api-access-jfkp5\") pod \"node-resolver-7rprf\" (UID: \"e0307f2d-fa2f-42dd-984e-148f44510d07\") " pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122966 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-cnibin\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-os-release\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.187132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-var-lib-cni-multus\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f97c53-b5ae-4a60-887a-eedcc84baf37-ovnkube-config\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123036 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-sys\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123080 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-run-systemd\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzml\" (UniqueName: \"kubernetes.io/projected/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-kube-api-access-wjzml\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-sysctl-conf\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123132 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/522a5dd5-0b75-4721-830e-42417948a408-dbus\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123166 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b223bb6-1dec-46f4-a6a8-7204b718d631-serviceca\") pod \"node-ca-lgdc4\" (UID: \"7b223bb6-1dec-46f4-a6a8-7204b718d631\") " pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0307f2d-fa2f-42dd-984e-148f44510d07-tmp-dir\") pod \"node-resolver-7rprf\" (UID: \"e0307f2d-fa2f-42dd-984e-148f44510d07\") " pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123206 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-log-socket\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123393 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f97c53-b5ae-4a60-887a-eedcc84baf37-env-overrides\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123499 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-system-cni-dir\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-run\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123553 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-etc-selinux\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.266186 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-kubernetes\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123641 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d502afb-2fd3-4687-a99c-ffb76d684ea6-iptables-alerter-script\") pod \"iptables-alerter-4zhwl\" (UID: \"1d502afb-2fd3-4687-a99c-ffb76d684ea6\") " pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-host-cni-netd\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123781 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0307f2d-fa2f-42dd-984e-148f44510d07-hosts-file\") pod \"node-resolver-7rprf\" (UID: \"e0307f2d-fa2f-42dd-984e-148f44510d07\") " pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123816 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-systemd-units\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.123918 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-lib-modules\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/096a094e-c379-445d-8e2f-5eb81a93f05a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124130 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" event={"ID":"3d3debe7f1b24cd0f146d976ae1a3f64","Type":"ContainerStarted","Data":"fdadd02f15d922371792d58ebdde3dfbab076212db667c1f877b5b58cc628bbd"} Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b223bb6-1dec-46f4-a6a8-7204b718d631-host\") pod \"node-ca-lgdc4\" (UID: \"7b223bb6-1dec-46f4-a6a8-7204b718d631\") " pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-run-openvswitch\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.122666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-sys-fs\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124258 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/096a094e-c379-445d-8e2f-5eb81a93f05a-cnibin\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124285 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-sys\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124309 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-var-lib-kubelet\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.124317 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.124384 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret podName:522a5dd5-0b75-4721-830e-42417948a408 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:30.62436525 +0000 UTC m=+3.046589578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret") pod "global-pull-secret-syncer-ffhs6" (UID: "522a5dd5-0b75-4721-830e-42417948a408") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124432 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f97c53-b5ae-4a60-887a-eedcc84baf37-log-socket\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-sysctl-conf\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.267057 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124563 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0307f2d-fa2f-42dd-984e-148f44510d07-tmp-dir\") pod \"node-resolver-7rprf\" (UID: \"e0307f2d-fa2f-42dd-984e-148f44510d07\") " pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124601 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/522a5dd5-0b75-4721-830e-42417948a408-dbus\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.124812 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-tmp\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.127491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/300ecf05-298e-4db7-b26d-f7b7abeb1855-agent-certs\") pod \"konnectivity-agent-cdbwf\" (UID: \"300ecf05-298e-4db7-b26d-f7b7abeb1855\") " pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.128899 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"85c349124b4cc1127652c4cd0349da6a","Type":"ContainerStarted","Data":"e4ab016fb548536cd5d14199780fa5217ca7f1effad780cdcc4b241c9c2aa1c4"} Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.129430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f97c53-b5ae-4a60-887a-eedcc84baf37-ovn-node-metrics-cert\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.130561 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-etc-tuned\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.131030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jxh\" (UniqueName: \"kubernetes.io/projected/1d502afb-2fd3-4687-a99c-ffb76d684ea6-kube-api-access-r2jxh\") pod \"iptables-alerter-4zhwl\" (UID: \"1d502afb-2fd3-4687-a99c-ffb76d684ea6\") " pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.137068 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6796\" (UniqueName: \"kubernetes.io/projected/c4f97c53-b5ae-4a60-887a-eedcc84baf37-kube-api-access-q6796\") pod \"ovnkube-node-w4xw9\" (UID: \"c4f97c53-b5ae-4a60-887a-eedcc84baf37\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.137196 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fbww\" (UniqueName: \"kubernetes.io/projected/7b223bb6-1dec-46f4-a6a8-7204b718d631-kube-api-access-8fbww\") pod \"node-ca-lgdc4\" (UID: \"7b223bb6-1dec-46f4-a6a8-7204b718d631\") " pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.137292 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqlx\" (UniqueName: \"kubernetes.io/projected/4b67c3ba-3447-4520-a16a-83d7c6fe5fce-kube-api-access-fgqlx\") pod \"tuned-mfsm5\" (UID: \"4b67c3ba-3447-4520-a16a-83d7c6fe5fce\") " pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.137804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkp5\" (UniqueName: \"kubernetes.io/projected/e0307f2d-fa2f-42dd-984e-148f44510d07-kube-api-access-jfkp5\") pod \"node-resolver-7rprf\" (UID: \"e0307f2d-fa2f-42dd-984e-148f44510d07\") " pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.138534 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5rft\" (UniqueName: \"kubernetes.io/projected/096a094e-c379-445d-8e2f-5eb81a93f05a-kube-api-access-v5rft\") pod \"multus-additional-cni-plugins-qnjcn\" (UID: \"096a094e-c379-445d-8e2f-5eb81a93f05a\") " pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.138816 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzml\" (UniqueName: \"kubernetes.io/projected/71f3b91e-9ef9-4b79-aeb2-4f2babeacca5-kube-api-access-wjzml\") pod \"aws-ebs-csi-driver-node-xz857\" (UID: \"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-cni-dir\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-var-lib-kubelet\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.269254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-etc-kubernetes\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nct6\" (UniqueName: \"kubernetes.io/projected/82fc003a-678c-4e3d-90ab-0c64f693f817-kube-api-access-7nct6\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-conf-dir\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-cni-dir\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227145 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-hostroot\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227135 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-var-lib-kubelet\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227182 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-hostroot\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227202 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-etc-kubernetes\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-conf-dir\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-socket-dir-parent\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnp9j\" (UniqueName: \"kubernetes.io/projected/a756c7f8-fc87-4f5b-9d5c-803a4d580084-kube-api-access-rnp9j\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227280 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-socket-dir-parent\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227304 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-run-k8s-cni-cncf-io\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-run-netns\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a756c7f8-fc87-4f5b-9d5c-803a4d580084-cni-binary-copy\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-run-k8s-cni-cncf-io\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227383 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-daemon-config\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227388 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-run-netns\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.270297 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-system-cni-dir\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-run-multus-certs\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227456 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-var-lib-cni-bin\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-var-lib-cni-bin\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227496 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-run-multus-certs\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227531 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-system-cni-dir\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-cnibin\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227561 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-os-release\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-cnibin\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227581 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-var-lib-cni-multus\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-os-release\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227652 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a756c7f8-fc87-4f5b-9d5c-803a4d580084-host-var-lib-cni-multus\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.227680 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.227729 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:30.727713099 +0000 UTC m=+3.149937413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a756c7f8-fc87-4f5b-9d5c-803a4d580084-cni-binary-copy\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.227965 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a756c7f8-fc87-4f5b-9d5c-803a4d580084-multus-daemon-config\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.237224 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.237246 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:36:30.271166 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.237305 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sqdtk for pod openshift-network-diagnostics/network-check-target-5dnnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:30.272655 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.237377 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk podName:da54353b-b322-4ed2-b608-ebcd592f4922 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:30.737353283 +0000 UTC m=+3.159577612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sqdtk" (UniqueName: "kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk") pod "network-check-target-5dnnv" (UID: "da54353b-b322-4ed2-b608-ebcd592f4922") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:30.272655 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.239503 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnp9j\" (UniqueName: \"kubernetes.io/projected/a756c7f8-fc87-4f5b-9d5c-803a4d580084-kube-api-access-rnp9j\") pod \"multus-cxvqm\" (UID: \"a756c7f8-fc87-4f5b-9d5c-803a4d580084\") " pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.272655 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.239643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nct6\" (UniqueName: \"kubernetes.io/projected/82fc003a-678c-4e3d-90ab-0c64f693f817-kube-api-access-7nct6\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:30.356414 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.299110 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" Jul 19 14:36:30.356414 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.306191 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:30.356414 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.323628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4zhwl" Jul 19 14:36:30.356414 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.330237 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:30.356414 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.341215 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" Jul 19 14:36:30.356414 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:30.342576 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d502afb_2fd3_4687_a99c_ffb76d684ea6.slice/crio-31083d0d878c89174e860950493981727646a95d60764268ebae853f023b97fd WatchSource:0}: Error finding container 31083d0d878c89174e860950493981727646a95d60764268ebae853f023b97fd: Status 404 returned error can't find the container with id 31083d0d878c89174e860950493981727646a95d60764268ebae853f023b97fd Jul 19 14:36:30.356414 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.348558 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lgdc4" Jul 19 14:36:30.356414 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:30.349667 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f97c53_b5ae_4a60_887a_eedcc84baf37.slice/crio-c68485599d66133d1fa4d2b937e2998920609b1144b5c670ab1711eaad4fade7 WatchSource:0}: Error finding container c68485599d66133d1fa4d2b937e2998920609b1144b5c670ab1711eaad4fade7: Status 404 returned error can't find the container with id c68485599d66133d1fa4d2b937e2998920609b1144b5c670ab1711eaad4fade7 Jul 19 14:36:30.357671 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.357654 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7rprf" Jul 19 14:36:30.359698 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:30.359654 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f3b91e_9ef9_4b79_aeb2_4f2babeacca5.slice/crio-f5e1e53c49252b16e9609b578da50ddf13d76983eab9904c938c5e662cabc60e WatchSource:0}: Error finding container f5e1e53c49252b16e9609b578da50ddf13d76983eab9904c938c5e662cabc60e: Status 404 returned error can't find the container with id f5e1e53c49252b16e9609b578da50ddf13d76983eab9904c938c5e662cabc60e Jul 19 14:36:30.365473 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.364264 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" Jul 19 14:36:30.371147 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.371129 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cxvqm" Jul 19 14:36:30.374943 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:30.374851 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b223bb6_1dec_46f4_a6a8_7204b718d631.slice/crio-64dbc2d36c800147ac3420a0095820faa0ce6cd67f88199be7fa9104d083c33b WatchSource:0}: Error finding container 64dbc2d36c800147ac3420a0095820faa0ce6cd67f88199be7fa9104d083c33b: Status 404 returned error can't find the container with id 64dbc2d36c800147ac3420a0095820faa0ce6cd67f88199be7fa9104d083c33b Jul 19 14:36:30.376733 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:30.376711 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0307f2d_fa2f_42dd_984e_148f44510d07.slice/crio-27f69dbeafca396612dec33fc77c0a9b39c3713717d5d2b4d123fe1258d3a269 WatchSource:0}: Error finding container 27f69dbeafca396612dec33fc77c0a9b39c3713717d5d2b4d123fe1258d3a269: Status 404 returned error can't find the container with id 27f69dbeafca396612dec33fc77c0a9b39c3713717d5d2b4d123fe1258d3a269 Jul 19 14:36:30.388973 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:30.385255 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096a094e_c379_445d_8e2f_5eb81a93f05a.slice/crio-d28d1028f18b10364d2f229cc1dbd9b5ff63baef979569b7a7e9ac7030ef7be4 WatchSource:0}: Error finding container d28d1028f18b10364d2f229cc1dbd9b5ff63baef979569b7a7e9ac7030ef7be4: Status 404 returned error can't find the container with id d28d1028f18b10364d2f229cc1dbd9b5ff63baef979569b7a7e9ac7030ef7be4 Jul 19 14:36:30.388973 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:36:30.385739 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda756c7f8_fc87_4f5b_9d5c_803a4d580084.slice/crio-1e574ebbbaed8fde766d2895e54352787c3b9cdc533f7841341e2f740df6d401 WatchSource:0}: Error finding container 1e574ebbbaed8fde766d2895e54352787c3b9cdc533f7841341e2f740df6d401: Status 404 returned error can't find the container with id 1e574ebbbaed8fde766d2895e54352787c3b9cdc533f7841341e2f740df6d401 Jul 19 14:36:30.630385 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.630345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:30.630547 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.630514 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:30.630613 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.630583 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret podName:522a5dd5-0b75-4721-830e-42417948a408 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:31.63056271 +0000 UTC m=+4.052787035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret") pod "global-pull-secret-syncer-ffhs6" (UID: "522a5dd5-0b75-4721-830e-42417948a408") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:30.731159 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.731126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:30.731339 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.731289 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:30.731394 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.731355 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:31.731340822 +0000 UTC m=+4.153565136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:30.832202 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:30.832167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:30.832375 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.832355 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:36:30.832441 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.832379 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:36:30.832441 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.832394 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sqdtk for pod openshift-network-diagnostics/network-check-target-5dnnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:30.832540 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:30.832458 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk podName:da54353b-b322-4ed2-b608-ebcd592f4922 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:31.832436214 +0000 UTC m=+4.254660538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqdtk" (UniqueName: "kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk") pod "network-check-target-5dnnv" (UID: "da54353b-b322-4ed2-b608-ebcd592f4922") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:31.067103 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.067012 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 14:31:30 +0000 UTC" deadline="2027-12-15 01:19:22.67602927 +0000 UTC" Jul 19 14:36:31.067103 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.067056 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12322h42m51.608978306s" Jul 19 14:36:31.102891 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.101622 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:31.102891 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:31.101765 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:31.132460 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.132383 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" event={"ID":"096a094e-c379-445d-8e2f-5eb81a93f05a","Type":"ContainerStarted","Data":"d28d1028f18b10364d2f229cc1dbd9b5ff63baef979569b7a7e9ac7030ef7be4"} Jul 19 14:36:31.133610 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.133571 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lgdc4" event={"ID":"7b223bb6-1dec-46f4-a6a8-7204b718d631","Type":"ContainerStarted","Data":"64dbc2d36c800147ac3420a0095820faa0ce6cd67f88199be7fa9104d083c33b"} Jul 19 14:36:31.135280 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.135255 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" event={"ID":"c4f97c53-b5ae-4a60-887a-eedcc84baf37","Type":"ContainerStarted","Data":"c68485599d66133d1fa4d2b937e2998920609b1144b5c670ab1711eaad4fade7"} Jul 19 14:36:31.136601 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.136577 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cdbwf" event={"ID":"300ecf05-298e-4db7-b26d-f7b7abeb1855","Type":"ContainerStarted","Data":"c468a948e014fa9dad5c9d4dd2d659f58537b564a0a4671d6b052192001ba870"} Jul 19 14:36:31.137848 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.137788 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cxvqm" event={"ID":"a756c7f8-fc87-4f5b-9d5c-803a4d580084","Type":"ContainerStarted","Data":"1e574ebbbaed8fde766d2895e54352787c3b9cdc533f7841341e2f740df6d401"} Jul 19 14:36:31.139069 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.139045 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7rprf" event={"ID":"e0307f2d-fa2f-42dd-984e-148f44510d07","Type":"ContainerStarted","Data":"27f69dbeafca396612dec33fc77c0a9b39c3713717d5d2b4d123fe1258d3a269"} Jul 19 14:36:31.141065 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.141038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" event={"ID":"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5","Type":"ContainerStarted","Data":"f5e1e53c49252b16e9609b578da50ddf13d76983eab9904c938c5e662cabc60e"} Jul 19 14:36:31.143414 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.143390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4zhwl" event={"ID":"1d502afb-2fd3-4687-a99c-ffb76d684ea6","Type":"ContainerStarted","Data":"31083d0d878c89174e860950493981727646a95d60764268ebae853f023b97fd"} Jul 19 14:36:31.144907 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.144843 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" event={"ID":"4b67c3ba-3447-4520-a16a-83d7c6fe5fce","Type":"ContainerStarted","Data":"32620070f81ba018fd15a7d3eec1499d2f73e3368aae42634e05f86acbbbc828"} Jul 19 14:36:31.639120 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.638464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:31.639120 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:31.638677 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:31.639120 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:31.638745 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret podName:522a5dd5-0b75-4721-830e-42417948a408 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:33.638725292 +0000 UTC m=+6.060949604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret") pod "global-pull-secret-syncer-ffhs6" (UID: "522a5dd5-0b75-4721-830e-42417948a408") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:31.739655 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.739061 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:31.739655 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:31.739273 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:31.739655 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:31.739335 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:33.739316049 +0000 UTC m=+6.161540361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:31.840512 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:31.839897 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:31.840512 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:31.840050 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:36:31.840512 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:31.840069 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:36:31.840512 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:31.840081 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sqdtk for pod openshift-network-diagnostics/network-check-target-5dnnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:31.840512 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:31.840139 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk podName:da54353b-b322-4ed2-b608-ebcd592f4922 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:33.840122364 +0000 UTC m=+6.262346675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqdtk" (UniqueName: "kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk") pod "network-check-target-5dnnv" (UID: "da54353b-b322-4ed2-b608-ebcd592f4922") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:32.067975 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:32.067817 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 14:31:30 +0000 UTC" deadline="2028-02-23 11:50:19.69298039 +0000 UTC" Jul 19 14:36:32.067975 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:32.067891 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14013h13m47.625094108s" Jul 19 14:36:32.101571 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:32.101526 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:32.101741 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:32.101656 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:32.102012 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:32.101525 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:32.102165 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:32.102124 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:33.101743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:33.101279 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:33.101743 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:33.101406 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:33.655822 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:33.655783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:33.655997 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:33.655985 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:33.656052 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:33.656045 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret podName:522a5dd5-0b75-4721-830e-42417948a408 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:37.65602729 +0000 UTC m=+10.078251615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret") pod "global-pull-secret-syncer-ffhs6" (UID: "522a5dd5-0b75-4721-830e-42417948a408") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:33.756789 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:33.756640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:33.756950 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:33.756799 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:33.756950 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:33.756910 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:37.756856525 +0000 UTC m=+10.179080851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:33.858063 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:33.857374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:33.858063 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:33.857592 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:36:33.858063 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:33.857613 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:36:33.858063 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:33.857627 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sqdtk for pod openshift-network-diagnostics/network-check-target-5dnnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:33.858063 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:33.857686 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk podName:da54353b-b322-4ed2-b608-ebcd592f4922 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:37.857667713 +0000 UTC m=+10.279892025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqdtk" (UniqueName: "kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk") pod "network-check-target-5dnnv" (UID: "da54353b-b322-4ed2-b608-ebcd592f4922") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:34.101123 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:34.101081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:34.101291 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:34.101216 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:34.101378 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:34.101327 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:34.101470 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:34.101447 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:35.101729 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:35.101699 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:35.102371 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:35.101828 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:36.101582 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:36.101418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:36.101582 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:36.101439 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:36.101582 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:36.101549 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:36.102252 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:36.101652 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:37.101907 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:37.101525 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:37.101907 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:37.101665 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:37.688923 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:37.688883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:37.689102 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:37.689053 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:37.689219 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:37.689205 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret podName:522a5dd5-0b75-4721-830e-42417948a408 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:45.689147436 +0000 UTC m=+18.111371759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret") pod "global-pull-secret-syncer-ffhs6" (UID: "522a5dd5-0b75-4721-830e-42417948a408") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:37.790147 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:37.789897 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:37.790147 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:37.790098 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:37.790369 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:37.790166 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:45.790148245 +0000 UTC m=+18.212372560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:37.891304 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:37.891173 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:37.891497 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:37.891356 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:36:37.891497 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:37.891374 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:36:37.891497 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:37.891388 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sqdtk for pod openshift-network-diagnostics/network-check-target-5dnnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:37.891497 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:37.891452 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk podName:da54353b-b322-4ed2-b608-ebcd592f4922 nodeName:}" failed. No retries permitted until 2026-07-19 14:36:45.891433484 +0000 UTC m=+18.313657799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqdtk" (UniqueName: "kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk") pod "network-check-target-5dnnv" (UID: "da54353b-b322-4ed2-b608-ebcd592f4922") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:38.102974 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:38.102684 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:38.102974 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:38.102820 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:38.102974 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:38.102878 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:38.103528 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:38.102989 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:39.101905 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:39.101858 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:39.102091 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:39.102011 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:40.101105 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:40.101069 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:40.101528 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:40.101210 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:40.101528 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:40.101279 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:40.101528 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:40.101405 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:41.100812 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:41.100770 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:41.100995 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:41.100908 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:42.101531 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:42.101489 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:42.102005 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:42.101640 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:42.102005 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:42.101665 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:42.105143 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:42.102692 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:43.101667 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:43.101637 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:43.102091 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:43.101738 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:44.101092 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:44.101052 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:44.101286 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:44.101056 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:44.101286 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:44.101185 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:44.101394 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:44.101291 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:45.100804 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:45.100774 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:45.101236 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:45.100896 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:45.751260 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:45.751205 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:45.751431 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:45.751381 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:45.751478 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:45.751461 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret podName:522a5dd5-0b75-4721-830e-42417948a408 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:01.751440907 +0000 UTC m=+34.173665219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret") pod "global-pull-secret-syncer-ffhs6" (UID: "522a5dd5-0b75-4721-830e-42417948a408") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:36:45.851998 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:45.851965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:45.852151 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:45.852090 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:45.852151 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:45.852146 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:01.852129579 +0000 UTC m=+34.274353888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:36:45.952351 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:45.952310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:45.952521 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:45.952478 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:36:45.952521 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:45.952511 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:36:45.952521 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:45.952524 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sqdtk for pod openshift-network-diagnostics/network-check-target-5dnnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:45.952637 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:45.952578 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk podName:da54353b-b322-4ed2-b608-ebcd592f4922 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:01.952564595 +0000 UTC m=+34.374788904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqdtk" (UniqueName: "kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk") pod "network-check-target-5dnnv" (UID: "da54353b-b322-4ed2-b608-ebcd592f4922") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:36:46.101496 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:46.101464 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:46.101951 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:46.101461 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:46.101951 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:46.101606 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:46.101951 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:46.101664 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:47.101492 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:47.101461 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:47.101641 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:47.101587 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:48.102614 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:48.102573 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:48.103060 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:48.102686 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:48.103060 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:48.102721 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:48.103060 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:48.102762 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:49.101056 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:49.100819 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:49.101187 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:49.101136 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:49.182087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:49.182031 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" event={"ID":"4b67c3ba-3447-4520-a16a-83d7c6fe5fce","Type":"ContainerStarted","Data":"a80d8c039ada22c3cc6e72cf4df6fa7c13159f924924a2bcd9f0c80997c939db"} Jul 19 14:36:49.183549 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:49.183524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" event={"ID":"3d3debe7f1b24cd0f146d976ae1a3f64","Type":"ContainerStarted","Data":"9b095dc1c933579426f6964f6859e3280745ef245fde4f626ee9a740bd74cdf9"} Jul 19 14:36:49.184753 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:49.184732 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cxvqm" event={"ID":"a756c7f8-fc87-4f5b-9d5c-803a4d580084","Type":"ContainerStarted","Data":"22450906bda5b99ed82ab35d2d078cdb4f5c27680b20887ef0d349fca3de7a74"} Jul 19 14:36:49.197212 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:49.197166 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mfsm5" podStartSLOduration=2.669294983 podStartE2EDuration="21.197152249s" totalImagesPullingTime="18.527857266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:49.196811535 +0000 UTC m=+21.619035865" watchObservedRunningTime="2026-07-19 14:36:49.197152249 +0000 UTC m=+21.619376577" Jul 19 14:36:49.227296 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:49.227251 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" podStartSLOduration=20.227232876 podStartE2EDuration="20.227232876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:49.226716709 +0000 UTC m=+21.648941041" watchObservedRunningTime="2026-07-19 14:36:49.227232876 +0000 UTC m=+21.649457207" Jul 19 14:36:49.227474 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:49.227428 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cxvqm" podStartSLOduration=2.727920831 podStartE2EDuration="21.227423946s" totalImagesPullingTime="18.499503115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:49.210999846 +0000 UTC m=+21.633224194" watchObservedRunningTime="2026-07-19 14:36:49.227423946 +0000 UTC m=+21.649648272" Jul 19 14:36:50.101550 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.101373 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:50.101659 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.101443 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:50.101659 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:50.101628 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:50.101750 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:50.101730 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:50.188391 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.188352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7rprf" event={"ID":"e0307f2d-fa2f-42dd-984e-148f44510d07","Type":"ContainerStarted","Data":"1aa6458743fcb32fa4133c46197257853131fe034035f74c086191698a6e4d0e"} Jul 19 14:36:50.190298 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.190271 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" event={"ID":"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5","Type":"ContainerStarted","Data":"9254e53390ed48e54c75acf7941b6763854e14dd974d661d6b6818b7a6374c9f"} Jul 19 14:36:50.192052 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.192026 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4zhwl" event={"ID":"1d502afb-2fd3-4687-a99c-ffb76d684ea6","Type":"ContainerStarted","Data":"06f72c48749a735a1a586e692daee4329c8944658bbf45e3604ced2169a92a5a"} Jul 19 14:36:50.193570 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.193546 2559 generic.go:356] "Generic (PLEG): container finished" podID="85c349124b4cc1127652c4cd0349da6a" containerID="a50dae95923ed4a9be5f447e5530bca60bfd932dcbd75aa3a479c859ad278dd0" exitCode=0 Jul 19 14:36:50.193666 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.193625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"85c349124b4cc1127652c4cd0349da6a","Type":"ContainerDied","Data":"a50dae95923ed4a9be5f447e5530bca60bfd932dcbd75aa3a479c859ad278dd0"} Jul 19 14:36:50.195376 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.195352 2559 generic.go:356] "Generic (PLEG): container finished" podID="096a094e-c379-445d-8e2f-5eb81a93f05a" containerID="855537a74ef893788765c44b187f79f67b7bc8c9c790dbcb934254b748258541" exitCode=0 Jul 19 14:36:50.195463 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.195423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" event={"ID":"096a094e-c379-445d-8e2f-5eb81a93f05a","Type":"ContainerDied","Data":"855537a74ef893788765c44b187f79f67b7bc8c9c790dbcb934254b748258541"} Jul 19 14:36:50.196849 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.196824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lgdc4" event={"ID":"7b223bb6-1dec-46f4-a6a8-7204b718d631","Type":"ContainerStarted","Data":"90e02b946072dbd7a778458258c3d666a588e49715b86df423a5db897d8739ac"} Jul 19 14:36:50.200204 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.200183 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" event={"ID":"c4f97c53-b5ae-4a60-887a-eedcc84baf37","Type":"ContainerStarted","Data":"a698111985d80b54223b166e9475ad99b4e085d3022f030d7caf3ee8259f3658"} Jul 19 14:36:50.200300 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.200256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" event={"ID":"c4f97c53-b5ae-4a60-887a-eedcc84baf37","Type":"ContainerStarted","Data":"523820975ccec74e65876be05dfea97b926d2a20fddd23557daafa688f9b1f3f"} Jul 19 14:36:50.200361 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.200305 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" event={"ID":"c4f97c53-b5ae-4a60-887a-eedcc84baf37","Type":"ContainerStarted","Data":"49b721ba2a0a40a384115ace5ec0ebe0f42dbb0bfdfa9b66d543ce1dbd5e5a52"} Jul 19 14:36:50.200361 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.200354 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" event={"ID":"c4f97c53-b5ae-4a60-887a-eedcc84baf37","Type":"ContainerStarted","Data":"78d3997559b8538748f4ed39887ca65e00bff7702919212ea994cb6ceaf79024"} Jul 19 14:36:50.200442 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.200399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" event={"ID":"c4f97c53-b5ae-4a60-887a-eedcc84baf37","Type":"ContainerStarted","Data":"ba242598f22f55691fc4958172310d636a116f29f060a8244c642e2df6e2a6d4"} Jul 19 14:36:50.200488 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.200443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" event={"ID":"c4f97c53-b5ae-4a60-887a-eedcc84baf37","Type":"ContainerStarted","Data":"71edab8828122e06eee6d7edc1468ce328b0144a39f79aceb2bc2a5919e10ade"} Jul 19 14:36:50.201733 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.201703 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cdbwf" event={"ID":"300ecf05-298e-4db7-b26d-f7b7abeb1855","Type":"ContainerStarted","Data":"f2b617b6f830f1c40e9de9171433c0882d77a5abd2de53e3dd3cba0ef6afa0eb"} Jul 19 14:36:50.202984 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.202939 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7rprf" podStartSLOduration=3.734617019 podStartE2EDuration="22.202926337s" totalImagesPullingTime="18.468309318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:50.202110729 +0000 UTC m=+22.624335065" watchObservedRunningTime="2026-07-19 14:36:50.202926337 +0000 UTC m=+22.625150671" Jul 19 14:36:50.215594 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.215562 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cdbwf" podStartSLOduration=3.824512144 podStartE2EDuration="22.215551241s" totalImagesPullingTime="18.391039097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:50.214973467 +0000 UTC m=+22.637197801" watchObservedRunningTime="2026-07-19 14:36:50.215551241 +0000 UTC m=+22.637775572" Jul 19 14:36:50.275270 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.275106 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4zhwl" podStartSLOduration=3.888343625 podStartE2EDuration="22.275094142s" totalImagesPullingTime="18.386750517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:50.261727216 +0000 UTC m=+22.683951548" watchObservedRunningTime="2026-07-19 14:36:50.275094142 +0000 UTC m=+22.697318474" Jul 19 14:36:50.275478 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.275458 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lgdc4" podStartSLOduration=11.94993191 podStartE2EDuration="22.27545255s" totalImagesPullingTime="10.32552064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:50.274826229 +0000 UTC m=+22.697050562" watchObservedRunningTime="2026-07-19 14:36:50.27545255 +0000 UTC m=+22.697676881" Jul 19 14:36:50.759050 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.758984 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:50.759506 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:50.759491 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:51.028750 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.028727 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 19 14:36:51.089207 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.089094 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-19T14:36:51.028746611Z","UUID":"e86a4522-baa2-4a63-be7e-c6df82bc5a36","Handler":null,"Name":"","Endpoint":""} Jul 19 14:36:51.090693 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.090669 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 19 14:36:51.090822 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.090709 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 19 14:36:51.101547 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.101524 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:51.101689 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:51.101629 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:51.205689 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.205654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" event={"ID":"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5","Type":"ContainerStarted","Data":"9cd47abeb70243f43308fcc335f429430ab39b872cca2f9717bf681fe99185f4"} Jul 19 14:36:51.207574 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.207524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"85c349124b4cc1127652c4cd0349da6a","Type":"ContainerStarted","Data":"fe3c8fcc0854e33c0086c2d98ed4b2b85d912d9f0b8171fa19a8e0ea9edb6cc5"} Jul 19 14:36:51.208337 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.208316 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:51.208881 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.208848 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cdbwf" Jul 19 14:36:51.234360 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:51.234314 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" podStartSLOduration=22.234295829 podStartE2EDuration="22.234295829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:51.220926865 +0000 UTC m=+23.643151193" watchObservedRunningTime="2026-07-19 14:36:51.234295829 +0000 UTC m=+23.656520163" Jul 19 14:36:52.101432 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:52.101403 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:52.101608 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:52.101517 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:52.101608 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:52.101554 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:52.101720 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:52.101630 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:53.101225 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:53.101194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:53.101645 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:53.101297 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:53.214958 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:53.214923 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" event={"ID":"c4f97c53-b5ae-4a60-887a-eedcc84baf37","Type":"ContainerStarted","Data":"f9f76af45b91836efa8126061ba3161d3000e5226c3b557cb3207da76502755f"} Jul 19 14:36:53.216934 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:53.216910 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" event={"ID":"71f3b91e-9ef9-4b79-aeb2-4f2babeacca5","Type":"ContainerStarted","Data":"b4daed4eec2d279ed340405e6a61fe82308270d76b429e1ffbc6c5df87640a46"} Jul 19 14:36:53.236813 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:53.236766 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz857" podStartSLOduration=4.065476436 podStartE2EDuration="25.236753631s" totalImagesPullingTime="21.171277195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:53.235552656 +0000 UTC m=+25.657776988" watchObservedRunningTime="2026-07-19 14:36:53.236753631 +0000 UTC m=+25.658977960" Jul 19 14:36:54.100921 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:54.100889 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:54.101097 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:54.100889 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:54.101097 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:54.101019 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:54.101193 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:54.101091 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:55.100815 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:55.100785 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:55.101202 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:55.100912 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:56.101930 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:56.101732 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:56.102624 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:56.101732 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:56.102624 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:56.102034 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:56.102624 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:56.102106 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:56.225168 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:56.225138 2559 generic.go:356] "Generic (PLEG): container finished" podID="096a094e-c379-445d-8e2f-5eb81a93f05a" containerID="938f4ffb04dfee0b6cac679babf1271e398d34bf104a4ca2302f0dcedc4a18d1" exitCode=0 Jul 19 14:36:56.225318 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:56.225221 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" event={"ID":"096a094e-c379-445d-8e2f-5eb81a93f05a","Type":"ContainerDied","Data":"938f4ffb04dfee0b6cac679babf1271e398d34bf104a4ca2302f0dcedc4a18d1"} Jul 19 14:36:56.228995 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:56.228975 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" event={"ID":"c4f97c53-b5ae-4a60-887a-eedcc84baf37","Type":"ContainerStarted","Data":"73463c30d0a10f4fafe0128c529cd5e5d26d3ac6124bea05df112ce34f0ac77b"} Jul 19 14:36:56.229222 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:56.229206 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:56.244055 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:56.244032 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:56.271732 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:56.271690 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" podStartSLOduration=9.384305129 podStartE2EDuration="28.271679685s" totalImagesPullingTime="18.887374556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:36:56.271168662 +0000 UTC m=+28.693392993" watchObservedRunningTime="2026-07-19 14:36:56.271679685 +0000 UTC m=+28.693904016" Jul 19 14:36:57.101074 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.101037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:57.101211 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:57.101139 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:57.177224 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.176984 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5dnnv"] Jul 19 14:36:57.182149 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.181996 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ffhs6"] Jul 19 14:36:57.182292 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.182164 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:57.182292 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:57.182272 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:57.182740 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.182717 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zw8ds"] Jul 19 14:36:57.182852 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.182831 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:57.182979 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:57.182959 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:57.233651 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.233622 2559 generic.go:356] "Generic (PLEG): container finished" podID="096a094e-c379-445d-8e2f-5eb81a93f05a" containerID="0d3326239c0d032a1e479cabd3983ea4076df87d0e1d1143fa5fbe01dded9999" exitCode=0 Jul 19 14:36:57.233780 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.233697 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:57.233780 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.233697 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" event={"ID":"096a094e-c379-445d-8e2f-5eb81a93f05a","Type":"ContainerDied","Data":"0d3326239c0d032a1e479cabd3983ea4076df87d0e1d1143fa5fbe01dded9999"} Jul 19 14:36:57.234034 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.233998 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 19 14:36:57.234491 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.234476 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:57.236982 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:57.236954 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:36:57.249474 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:57.249454 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:58.110030 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:58.109986 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096a094e_c379_445d_8e2f_5eb81a93f05a.slice/crio-conmon-2ff12b4f3570828522b6c309e3762d521d306a4e80b37d766e0683215e4ac5d6.scope\": RecentStats: unable to find data in memory cache]" Jul 19 14:36:58.238428 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:58.238388 2559 generic.go:356] "Generic (PLEG): container finished" podID="096a094e-c379-445d-8e2f-5eb81a93f05a" containerID="2ff12b4f3570828522b6c309e3762d521d306a4e80b37d766e0683215e4ac5d6" exitCode=0 Jul 19 14:36:58.238802 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:58.238485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" event={"ID":"096a094e-c379-445d-8e2f-5eb81a93f05a","Type":"ContainerDied","Data":"2ff12b4f3570828522b6c309e3762d521d306a4e80b37d766e0683215e4ac5d6"} Jul 19 14:36:58.238802 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:58.238556 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 19 14:36:58.367541 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:58.367515 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:36:59.101419 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:59.101391 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:36:59.101419 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:59.101422 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:36:59.101632 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:36:59.101390 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:36:59.101632 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:59.101517 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:36:59.101728 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:59.101661 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:36:59.101770 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:36:59.101736 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:37:00.257173 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:00.257140 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4xw9" Jul 19 14:37:01.101113 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.101081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:37:01.101288 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.101081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:37:01.101288 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.101081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:37:01.101288 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.101242 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dnnv" podUID="da54353b-b322-4ed2-b608-ebcd592f4922" Jul 19 14:37:01.101420 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.101343 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:37:01.101458 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.101429 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffhs6" podUID="522a5dd5-0b75-4721-830e-42417948a408" Jul 19 14:37:01.774422 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.774386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:37:01.774895 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.774535 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:37:01.774895 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.774601 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret podName:522a5dd5-0b75-4721-830e-42417948a408 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:33.77458659 +0000 UTC m=+66.196810904 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret") pod "global-pull-secret-syncer-ffhs6" (UID: "522a5dd5-0b75-4721-830e-42417948a408") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:37:01.874994 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.874955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:37:01.875184 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.875081 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:37:01.875184 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.875152 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:33.875132873 +0000 UTC m=+66.297357230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:37:01.921489 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.921414 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeReady" Jul 19 14:37:01.921663 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.921562 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 19 14:37:01.954816 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.954777 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-666d9b4699-8pbzg"] Jul 19 14:37:01.975328 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.975289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:37:01.975498 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.975441 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:37:01.975498 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.975459 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:37:01.975498 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.975470 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sqdtk for pod openshift-network-diagnostics/network-check-target-5dnnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:37:01.975630 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:01.975526 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk podName:da54353b-b322-4ed2-b608-ebcd592f4922 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:33.975510459 +0000 UTC m=+66.397734782 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqdtk" (UniqueName: "kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk") pod "network-check-target-5dnnv" (UID: "da54353b-b322-4ed2-b608-ebcd592f4922") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:37:01.977437 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.977351 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bgdc6"] Jul 19 14:37:01.977567 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.977479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:01.979824 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.979800 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 19 14:37:01.980148 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.980119 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 19 14:37:01.980264 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.980129 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 19 14:37:01.980330 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.980266 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rz27h\"" Jul 19 14:37:01.987815 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.987790 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 19 14:37:01.991659 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.991636 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666d9b4699-8pbzg"] Jul 19 14:37:01.991763 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.991675 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9tkvk"] Jul 19 14:37:01.991950 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.991929 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:01.994227 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.994203 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 19 14:37:01.994323 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.994207 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 19 14:37:01.994323 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:01.994275 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8vqq7\"" Jul 19 14:37:02.003857 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.003838 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2"] Jul 19 14:37:02.003974 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.003961 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tkvk" Jul 19 14:37:02.006311 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.006164 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rbc88\"" Jul 19 14:37:02.006311 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.006196 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 19 14:37:02.006311 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.006226 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 19 14:37:02.006731 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.006706 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 19 14:37:02.018177 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.018150 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bgdc6"] Jul 19 14:37:02.018275 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.018190 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tkvk"] Jul 19 14:37:02.018275 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.018213 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2"] Jul 19 14:37:02.018275 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.018269 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:02.021126 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.021105 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 19 14:37:02.021228 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.021149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nfdzm\"" Jul 19 14:37:02.021228 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.021156 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 19 14:37:02.076226 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-image-registry-private-configuration\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.076226 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076236 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68cdeba4-359c-4409-8c6d-8297f763352b-ca-trust-extracted\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.076494 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-trusted-ca\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.076494 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076303 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3664d65b-c2f1-4254-a294-9f4123067a5b-config-volume\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.076494 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgfpd\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-kube-api-access-kgfpd\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.076494 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-installation-pull-secrets\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.076494 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076441 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3664d65b-c2f1-4254-a294-9f4123067a5b-tmp-dir\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.076494 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076467 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-registry-certificates\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.076494 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076490 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-bound-sa-token\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.076789 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.076789 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076559 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.076789 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.076588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtmb\" (UniqueName: \"kubernetes.io/projected/3664d65b-c2f1-4254-a294-9f4123067a5b-kube-api-access-mgtmb\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.177743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.177743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177698 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtmb\" (UniqueName: \"kubernetes.io/projected/3664d65b-c2f1-4254-a294-9f4123067a5b-kube-api-access-mgtmb\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.177743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177733 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-image-registry-private-configuration\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68cdeba4-359c-4409-8c6d-8297f763352b-ca-trust-extracted\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177777 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-trusted-ca\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177796 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177833 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3664d65b-c2f1-4254-a294-9f4123067a5b-config-volume\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgfpd\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-kube-api-access-kgfpd\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/017827df-343d-42db-beec-ce9b21b305f5-cert\") pod \"ingress-canary-9tkvk\" (UID: \"017827df-343d-42db-beec-ce9b21b305f5\") " pod="openshift-ingress-canary/ingress-canary-9tkvk" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.177981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-installation-pull-secrets\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.178044 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfcz\" (UniqueName: \"kubernetes.io/projected/017827df-343d-42db-beec-ce9b21b305f5-kube-api-access-2hfcz\") pod \"ingress-canary-9tkvk\" (UID: \"017827df-343d-42db-beec-ce9b21b305f5\") " pod="openshift-ingress-canary/ingress-canary-9tkvk" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.178090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3664d65b-c2f1-4254-a294-9f4123067a5b-tmp-dir\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.178121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.178116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-registry-certificates\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.178142 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-bound-sa-token\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.178194 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30c83ee5-7890-4240-957c-49de880b15a4-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.178223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.178251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68cdeba4-359c-4409-8c6d-8297f763352b-ca-trust-extracted\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.177832 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.178291 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d9b4699-8pbzg: secret "image-registry-tls" not found Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.178316 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.178343 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls podName:68cdeba4-359c-4409-8c6d-8297f763352b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:02.678324336 +0000 UTC m=+35.100548679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls") pod "image-registry-666d9b4699-8pbzg" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b") : secret "image-registry-tls" not found Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.178360 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls podName:3664d65b-c2f1-4254-a294-9f4123067a5b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:02.678350345 +0000 UTC m=+35.100574660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls") pod "dns-default-bgdc6" (UID: "3664d65b-c2f1-4254-a294-9f4123067a5b") : secret "dns-default-metrics-tls" not found Jul 19 14:37:02.178529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.178420 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3664d65b-c2f1-4254-a294-9f4123067a5b-config-volume\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.178967 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.178603 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3664d65b-c2f1-4254-a294-9f4123067a5b-tmp-dir\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.179301 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.179277 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-registry-certificates\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.179395 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.179277 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-trusted-ca\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.182417 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.182394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-installation-pull-secrets\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.182417 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.182404 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-image-registry-private-configuration\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.186328 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.186302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-bound-sa-token\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.186572 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.186553 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtmb\" (UniqueName: \"kubernetes.io/projected/3664d65b-c2f1-4254-a294-9f4123067a5b-kube-api-access-mgtmb\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.186641 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.186597 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgfpd\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-kube-api-access-kgfpd\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.279162 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.279130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/017827df-343d-42db-beec-ce9b21b305f5-cert\") pod \"ingress-canary-9tkvk\" (UID: \"017827df-343d-42db-beec-ce9b21b305f5\") " pod="openshift-ingress-canary/ingress-canary-9tkvk" Jul 19 14:37:02.279359 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.279196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfcz\" (UniqueName: \"kubernetes.io/projected/017827df-343d-42db-beec-ce9b21b305f5-kube-api-access-2hfcz\") pod \"ingress-canary-9tkvk\" (UID: \"017827df-343d-42db-beec-ce9b21b305f5\") " pod="openshift-ingress-canary/ingress-canary-9tkvk" Jul 19 14:37:02.279359 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.279243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30c83ee5-7890-4240-957c-49de880b15a4-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:02.279359 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.279297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:02.279513 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.279411 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:37:02.279513 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.279472 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:02.779451313 +0000 UTC m=+35.201675623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:37:02.279960 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.279931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30c83ee5-7890-4240-957c-49de880b15a4-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:02.281729 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.281710 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/017827df-343d-42db-beec-ce9b21b305f5-cert\") pod \"ingress-canary-9tkvk\" (UID: \"017827df-343d-42db-beec-ce9b21b305f5\") " pod="openshift-ingress-canary/ingress-canary-9tkvk" Jul 19 14:37:02.290273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.290248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfcz\" (UniqueName: \"kubernetes.io/projected/017827df-343d-42db-beec-ce9b21b305f5-kube-api-access-2hfcz\") pod \"ingress-canary-9tkvk\" (UID: \"017827df-343d-42db-beec-ce9b21b305f5\") " pod="openshift-ingress-canary/ingress-canary-9tkvk" Jul 19 14:37:02.313184 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.313159 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tkvk" Jul 19 14:37:02.681632 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.681598 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:02.681916 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.681644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:02.681916 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.681770 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:37:02.681916 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.681775 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:37:02.681916 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.681783 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d9b4699-8pbzg: secret "image-registry-tls" not found Jul 19 14:37:02.681916 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.681848 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls podName:3664d65b-c2f1-4254-a294-9f4123067a5b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:03.681827587 +0000 UTC m=+36.104051914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls") pod "dns-default-bgdc6" (UID: "3664d65b-c2f1-4254-a294-9f4123067a5b") : secret "dns-default-metrics-tls" not found Jul 19 14:37:02.681916 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.681892 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls podName:68cdeba4-359c-4409-8c6d-8297f763352b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:03.68188 +0000 UTC m=+36.104104323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls") pod "image-registry-666d9b4699-8pbzg" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b") : secret "image-registry-tls" not found Jul 19 14:37:02.783077 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.783036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:02.783451 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.783189 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:37:02.783451 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:02.783273 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:03.783256261 +0000 UTC m=+36.205480571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:37:02.939733 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:02.939670 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tkvk"] Jul 19 14:37:02.969074 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:37:02.969041 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017827df_343d_42db_beec_ce9b21b305f5.slice/crio-4f57455df7e198a2eb3214bb409e86e5a2ce4b1ca82a01d93f516bf87b74bc44 WatchSource:0}: Error finding container 4f57455df7e198a2eb3214bb409e86e5a2ce4b1ca82a01d93f516bf87b74bc44: Status 404 returned error can't find the container with id 4f57455df7e198a2eb3214bb409e86e5a2ce4b1ca82a01d93f516bf87b74bc44 Jul 19 14:37:03.101638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.101616 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:37:03.101740 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.101723 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:37:03.101820 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.101802 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:37:03.104115 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.104094 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 14:37:03.104553 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.104413 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 14:37:03.104553 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.104427 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 14:37:03.104553 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.104443 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xgfj\"" Jul 19 14:37:03.104553 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.104453 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5r6f\"" Jul 19 14:37:03.104553 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.104413 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 14:37:03.253237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.253198 2559 generic.go:356] "Generic (PLEG): container finished" podID="096a094e-c379-445d-8e2f-5eb81a93f05a" containerID="7ee0d44826689ba3fc820d2898d1c3229bccad7e051b2863d441f3473ae7c496" exitCode=0 Jul 19 14:37:03.253419 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.253284 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" event={"ID":"096a094e-c379-445d-8e2f-5eb81a93f05a","Type":"ContainerDied","Data":"7ee0d44826689ba3fc820d2898d1c3229bccad7e051b2863d441f3473ae7c496"} Jul 19 14:37:03.254323 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.254304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tkvk" event={"ID":"017827df-343d-42db-beec-ce9b21b305f5","Type":"ContainerStarted","Data":"4f57455df7e198a2eb3214bb409e86e5a2ce4b1ca82a01d93f516bf87b74bc44"} Jul 19 14:37:03.689572 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.689522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:03.689758 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.689581 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:03.689758 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:03.689692 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:37:03.689758 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:03.689707 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:37:03.689758 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:03.689719 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d9b4699-8pbzg: secret "image-registry-tls" not found Jul 19 14:37:03.689758 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:03.689757 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls podName:3664d65b-c2f1-4254-a294-9f4123067a5b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:05.689741898 +0000 UTC m=+38.111966207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls") pod "dns-default-bgdc6" (UID: "3664d65b-c2f1-4254-a294-9f4123067a5b") : secret "dns-default-metrics-tls" not found Jul 19 14:37:03.690060 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:03.689772 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls podName:68cdeba4-359c-4409-8c6d-8297f763352b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:05.689765736 +0000 UTC m=+38.111990046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls") pod "image-registry-666d9b4699-8pbzg" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b") : secret "image-registry-tls" not found Jul 19 14:37:03.790922 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:03.790822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:03.791718 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:03.790970 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:37:03.791718 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:03.791037 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:05.791021787 +0000 UTC m=+38.213246118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:37:04.260506 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:04.260469 2559 generic.go:356] "Generic (PLEG): container finished" podID="096a094e-c379-445d-8e2f-5eb81a93f05a" containerID="8fea228e7b10b9ffbd1c6ae2586f0e3594574a9a78770b1c6cd053e31e3ea626" exitCode=0 Jul 19 14:37:04.260699 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:04.260538 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" event={"ID":"096a094e-c379-445d-8e2f-5eb81a93f05a","Type":"ContainerDied","Data":"8fea228e7b10b9ffbd1c6ae2586f0e3594574a9a78770b1c6cd053e31e3ea626"} Jul 19 14:37:05.267061 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:05.267027 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" event={"ID":"096a094e-c379-445d-8e2f-5eb81a93f05a","Type":"ContainerStarted","Data":"7a060eb63d4b29c994a4c74d39ed34b46600e2e5e1eb0047c2f66fce188f4c01"} Jul 19 14:37:05.289226 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:05.289077 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qnjcn" podStartSLOduration=7.357883884 podStartE2EDuration="37.289059839s" totalImagesPullingTime="29.931175955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:37:05.287236436 +0000 UTC m=+37.709460768" watchObservedRunningTime="2026-07-19 14:37:05.289059839 +0000 UTC m=+37.711284179" Jul 19 14:37:05.707126 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:05.707100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:05.707249 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:05.707134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:05.707309 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:05.707257 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:37:05.707309 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:05.707270 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:37:05.707309 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:05.707281 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d9b4699-8pbzg: secret "image-registry-tls" not found Jul 19 14:37:05.707457 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:05.707316 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls podName:3664d65b-c2f1-4254-a294-9f4123067a5b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:09.707297189 +0000 UTC m=+42.129521506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls") pod "dns-default-bgdc6" (UID: "3664d65b-c2f1-4254-a294-9f4123067a5b") : secret "dns-default-metrics-tls" not found Jul 19 14:37:05.707457 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:05.707329 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls podName:68cdeba4-359c-4409-8c6d-8297f763352b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:09.707322886 +0000 UTC m=+42.129547195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls") pod "image-registry-666d9b4699-8pbzg" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b") : secret "image-registry-tls" not found Jul 19 14:37:05.808659 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:05.808104 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:05.808659 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:05.808283 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:37:05.808659 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:05.808344 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:09.808326086 +0000 UTC m=+42.230550403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:37:06.270694 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:06.270661 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tkvk" event={"ID":"017827df-343d-42db-beec-ce9b21b305f5","Type":"ContainerStarted","Data":"80fadf3bb852d320d6c5e9bd12eadc594881d517c6b728dd4217952bd57b2d12"} Jul 19 14:37:06.287074 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:06.287030 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9tkvk" podStartSLOduration=2.578868553 podStartE2EDuration="5.287016641s" totalImagesPullingTime="2.708148088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:37:06.286336986 +0000 UTC m=+38.708561318" watchObservedRunningTime="2026-07-19 14:37:06.287016641 +0000 UTC m=+38.709241004" Jul 19 14:37:09.740098 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:09.740060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:09.740448 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:09.740118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:09.740448 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:09.740227 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:37:09.740448 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:09.740272 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:37:09.740448 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:09.740288 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d9b4699-8pbzg: secret "image-registry-tls" not found Jul 19 14:37:09.740448 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:09.740299 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls podName:3664d65b-c2f1-4254-a294-9f4123067a5b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:17.740281235 +0000 UTC m=+50.162505548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls") pod "dns-default-bgdc6" (UID: "3664d65b-c2f1-4254-a294-9f4123067a5b") : secret "dns-default-metrics-tls" not found Jul 19 14:37:09.740448 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:09.740336 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls podName:68cdeba4-359c-4409-8c6d-8297f763352b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:17.740325438 +0000 UTC m=+50.162549748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls") pod "image-registry-666d9b4699-8pbzg" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b") : secret "image-registry-tls" not found Jul 19 14:37:09.841251 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:09.841219 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:09.841413 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:09.841372 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:37:09.841458 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:09.841429 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:17.841415583 +0000 UTC m=+50.263639892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:37:17.792241 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:17.792202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:17.792241 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:17.792243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:17.792683 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:17.792344 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:37:17.792683 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:17.792354 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d9b4699-8pbzg: secret "image-registry-tls" not found Jul 19 14:37:17.792683 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:17.792362 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:37:17.792683 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:17.792411 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls podName:68cdeba4-359c-4409-8c6d-8297f763352b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:33.792398165 +0000 UTC m=+66.214622474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls") pod "image-registry-666d9b4699-8pbzg" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b") : secret "image-registry-tls" not found Jul 19 14:37:17.792683 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:17.792422 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls podName:3664d65b-c2f1-4254-a294-9f4123067a5b nodeName:}" failed. No retries permitted until 2026-07-19 14:37:33.792417124 +0000 UTC m=+66.214641433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls") pod "dns-default-bgdc6" (UID: "3664d65b-c2f1-4254-a294-9f4123067a5b") : secret "dns-default-metrics-tls" not found Jul 19 14:37:17.893046 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:17.893003 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:17.893203 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:17.893161 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:37:17.893243 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:17.893225 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:37:33.893210088 +0000 UTC m=+66.315434397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:37:33.807529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:33.807485 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:37:33.808056 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:33.807554 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:37:33.808056 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:33.807615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:37:33.808056 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:33.807669 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:37:33.808056 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:33.807687 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d9b4699-8pbzg: secret "image-registry-tls" not found Jul 19 14:37:33.808056 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:33.807696 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:37:33.808056 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:33.807746 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls podName:3664d65b-c2f1-4254-a294-9f4123067a5b nodeName:}" failed. No retries permitted until 2026-07-19 14:38:05.807727447 +0000 UTC m=+98.229951770 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls") pod "dns-default-bgdc6" (UID: "3664d65b-c2f1-4254-a294-9f4123067a5b") : secret "dns-default-metrics-tls" not found Jul 19 14:37:33.808056 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:33.807760 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls podName:68cdeba4-359c-4409-8c6d-8297f763352b nodeName:}" failed. No retries permitted until 2026-07-19 14:38:05.807752732 +0000 UTC m=+98.229977041 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls") pod "image-registry-666d9b4699-8pbzg" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b") : secret "image-registry-tls" not found Jul 19 14:37:33.810321 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:33.810301 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 14:37:33.821301 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:33.821280 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/522a5dd5-0b75-4721-830e-42417948a408-original-pull-secret\") pod \"global-pull-secret-syncer-ffhs6\" (UID: \"522a5dd5-0b75-4721-830e-42417948a408\") " pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:37:33.908166 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:33.908135 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:37:33.908315 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:33.908181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:37:33.908315 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:33.908294 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:37:33.908385 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:33.908353 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:38:05.908340166 +0000 UTC m=+98.330564480 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:37:33.910577 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:33.910561 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 14:37:33.919143 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:33.919128 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 19 14:37:33.919184 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:37:33.919177 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:38:37.919163898 +0000 UTC m=+130.341388209 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : secret "metrics-daemon-secret" not found Jul 19 14:37:34.008666 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.008635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:37:34.011183 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.011164 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 14:37:34.021120 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.021103 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 14:37:34.025934 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.025918 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffhs6" Jul 19 14:37:34.041818 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.041797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqdtk\" (UniqueName: \"kubernetes.io/projected/da54353b-b322-4ed2-b608-ebcd592f4922-kube-api-access-sqdtk\") pod \"network-check-target-5dnnv\" (UID: \"da54353b-b322-4ed2-b608-ebcd592f4922\") " pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:37:34.179425 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.179396 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ffhs6"] Jul 19 14:37:34.182349 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:37:34.182317 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522a5dd5_0b75_4721_830e_42417948a408.slice/crio-3dc418d7f8fadbedba9c3c1182373f6d6894172417d1a187024e8657cac4d548 WatchSource:0}: Error finding container 3dc418d7f8fadbedba9c3c1182373f6d6894172417d1a187024e8657cac4d548: Status 404 returned error can't find the container with id 3dc418d7f8fadbedba9c3c1182373f6d6894172417d1a187024e8657cac4d548 Jul 19 14:37:34.317039 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.316987 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xgfj\"" Jul 19 14:37:34.325086 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.325068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:37:34.333905 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.333885 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ffhs6" event={"ID":"522a5dd5-0b75-4721-830e-42417948a408","Type":"ContainerStarted","Data":"3dc418d7f8fadbedba9c3c1182373f6d6894172417d1a187024e8657cac4d548"} Jul 19 14:37:34.435624 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:34.435590 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5dnnv"] Jul 19 14:37:34.438963 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:37:34.438933 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda54353b_b322_4ed2_b608_ebcd592f4922.slice/crio-183e844acd5f169c4acc19fec65ce51cd499f6a7930c5f43684b7c0778d74191 WatchSource:0}: Error finding container 183e844acd5f169c4acc19fec65ce51cd499f6a7930c5f43684b7c0778d74191: Status 404 returned error can't find the container with id 183e844acd5f169c4acc19fec65ce51cd499f6a7930c5f43684b7c0778d74191 Jul 19 14:37:35.337782 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:35.337740 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5dnnv" event={"ID":"da54353b-b322-4ed2-b608-ebcd592f4922","Type":"ContainerStarted","Data":"183e844acd5f169c4acc19fec65ce51cd499f6a7930c5f43684b7c0778d74191"} Jul 19 14:37:39.350664 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:39.350622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ffhs6" event={"ID":"522a5dd5-0b75-4721-830e-42417948a408","Type":"ContainerStarted","Data":"ec3758dfb5cb01875b7be8893a542298c5a3ee82adb3423440bc88b2d75e0718"} Jul 19 14:37:39.351940 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:39.351907 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5dnnv" event={"ID":"da54353b-b322-4ed2-b608-ebcd592f4922","Type":"ContainerStarted","Data":"7ff99e7f7e8aace1befd13e23d9eebfea36aff4dc74ca3e2836065cc1220f43a"} Jul 19 14:37:39.352055 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:39.351966 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:37:39.380444 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:39.380404 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ffhs6" podStartSLOduration=66.493124901 podStartE2EDuration="1m11.380392142s" totalImagesPullingTime="4.887267241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:37:39.365754678 +0000 UTC m=+71.787979010" watchObservedRunningTime="2026-07-19 14:37:39.380392142 +0000 UTC m=+71.802616451" Jul 19 14:37:39.380609 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:37:39.380590 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5dnnv" podStartSLOduration=66.751484284 podStartE2EDuration="1m11.380585924s" totalImagesPullingTime="4.62910164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:37:39.379830523 +0000 UTC m=+71.802054871" watchObservedRunningTime="2026-07-19 14:37:39.380585924 +0000 UTC m=+71.802810254" Jul 19 14:38:05.832809 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:38:05.832665 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:38:05.832809 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:38:05.832724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:38:05.833262 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:38:05.832829 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:38:05.833262 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:38:05.832832 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:38:05.833262 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:38:05.832932 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls podName:3664d65b-c2f1-4254-a294-9f4123067a5b nodeName:}" failed. No retries permitted until 2026-07-19 14:39:09.832909481 +0000 UTC m=+162.255133824 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls") pod "dns-default-bgdc6" (UID: "3664d65b-c2f1-4254-a294-9f4123067a5b") : secret "dns-default-metrics-tls" not found Jul 19 14:38:05.833262 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:38:05.832937 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d9b4699-8pbzg: secret "image-registry-tls" not found Jul 19 14:38:05.833262 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:38:05.832979 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls podName:68cdeba4-359c-4409-8c6d-8297f763352b nodeName:}" failed. No retries permitted until 2026-07-19 14:39:09.832968628 +0000 UTC m=+162.255192952 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls") pod "image-registry-666d9b4699-8pbzg" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b") : secret "image-registry-tls" not found Jul 19 14:38:05.933479 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:38:05.933446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:38:05.933624 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:38:05.933586 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:38:05.933661 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:38:05.933650 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:39:09.933635999 +0000 UTC m=+162.355860309 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:38:10.356143 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:38:10.356111 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5dnnv" Jul 19 14:38:37.957215 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:38:37.957180 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:38:37.957598 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:38:37.957296 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 19 14:38:37.957598 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:38:37.957355 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:40:39.957340176 +0000 UTC m=+252.379564485 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : secret "metrics-daemon-secret" not found Jul 19 14:39:04.992216 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:04.992161 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" podUID="68cdeba4-359c-4409-8c6d-8297f763352b" Jul 19 14:39:05.002354 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:05.002320 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bgdc6" podUID="3664d65b-c2f1-4254-a294-9f4123067a5b" Jul 19 14:39:05.028511 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:05.028475 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" podUID="30c83ee5-7890-4240-957c-49de880b15a4" Jul 19 14:39:05.549144 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:39:05.549113 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bgdc6" Jul 19 14:39:05.549314 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:39:05.549113 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:39:05.549314 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:39:05.549113 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:39:06.120883 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:06.120835 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:39:09.875530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:39:09.875470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:39:09.875530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:39:09.875530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:39:09.876109 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:09.875625 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:39:09.876109 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:09.875636 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:39:09.876109 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:09.875649 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d9b4699-8pbzg: secret "image-registry-tls" not found Jul 19 14:39:09.876109 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:09.875695 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls podName:68cdeba4-359c-4409-8c6d-8297f763352b nodeName:}" failed. No retries permitted until 2026-07-19 14:41:11.875682292 +0000 UTC m=+284.297906602 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls") pod "image-registry-666d9b4699-8pbzg" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b") : secret "image-registry-tls" not found Jul 19 14:39:09.876109 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:09.875708 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls podName:3664d65b-c2f1-4254-a294-9f4123067a5b nodeName:}" failed. No retries permitted until 2026-07-19 14:41:11.875702318 +0000 UTC m=+284.297926627 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls") pod "dns-default-bgdc6" (UID: "3664d65b-c2f1-4254-a294-9f4123067a5b") : secret "dns-default-metrics-tls" not found Jul 19 14:39:09.976317 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:39:09.976281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:39:09.976462 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:09.976420 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:39:09.976530 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:39:09.976481 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:41:11.976467987 +0000 UTC m=+284.398692299 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:39:21.101805 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:39:21.101704 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:40:39.991754 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:39.991700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:40:39.992310 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:39.991836 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 19 14:40:39.992310 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:39.991933 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs podName:82fc003a-678c-4e3d-90ab-0c64f693f817 nodeName:}" failed. No retries permitted until 2026-07-19 14:42:41.991909655 +0000 UTC m=+374.414133968 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs") pod "network-metrics-daemon-zw8ds" (UID: "82fc003a-678c-4e3d-90ab-0c64f693f817") : secret "metrics-daemon-secret" not found Jul 19 14:40:55.650636 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.650557 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-sjhrl"] Jul 19 14:40:55.652263 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.652247 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.654881 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.654847 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-74gwb"] Jul 19 14:40:55.655505 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.655480 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 19 14:40:55.655621 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.655515 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 19 14:40:55.655621 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.655561 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 19 14:40:55.655621 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.655519 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 19 14:40:55.656373 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.656352 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.659339 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.659318 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 19 14:40:55.660042 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.660019 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf"] Jul 19 14:40:55.660141 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.660117 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 19 14:40:55.660356 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.660338 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:40:55.660927 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.660888 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-kzvfq\"" Jul 19 14:40:55.661087 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.661061 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 19 14:40:55.661455 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.661436 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 19 14:40:55.661553 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.661538 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:55.663432 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.663410 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9zpqv\"" Jul 19 14:40:55.670404 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.670382 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 19 14:40:55.670597 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.670576 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 19 14:40:55.670700 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.670409 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-52jts\"" Jul 19 14:40:55.670758 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.670447 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 19 14:40:55.671419 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.671070 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-sjhrl"] Jul 19 14:40:55.671643 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.671622 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 19 14:40:55.672688 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.672668 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 19 14:40:55.676669 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.676652 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-74gwb"] Jul 19 14:40:55.684936 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.684908 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf"] Jul 19 14:40:55.696574 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696554 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-serving-cert\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.696661 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696579 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-serving-cert\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.696661 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696597 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.696661 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696615 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-service-ca-bundle\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.696780 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696688 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:55.696780 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-tmp\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.696780 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-snapshots\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.696904 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696798 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggj5z\" (UniqueName: \"kubernetes.io/projected/b8785e97-0280-4335-a04c-823904ee8e2f-kube-api-access-ggj5z\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:55.696904 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696819 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b8785e97-0280-4335-a04c-823904ee8e2f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:55.696904 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkr7t\" (UniqueName: \"kubernetes.io/projected/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-kube-api-access-qkr7t\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.696904 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-config\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.696904 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696889 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-trusted-ca\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.697069 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.696906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv9zl\" (UniqueName: \"kubernetes.io/projected/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-kube-api-access-tv9zl\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.741502 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.741474 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r"] Jul 19 14:40:55.743583 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.743567 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r" Jul 19 14:40:55.745966 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.745942 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rljm2\"" Jul 19 14:40:55.747220 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.747198 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb"] Jul 19 14:40:55.749324 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.749306 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66bd9878b6-whfdj"] Jul 19 14:40:55.749413 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.749397 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.751172 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.751154 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.751802 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.751784 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 19 14:40:55.751802 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.751795 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:40:55.751991 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.751815 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 19 14:40:55.751991 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.751842 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 19 14:40:55.751991 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.751943 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xmtnx\"" Jul 19 14:40:55.753926 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.753905 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 19 14:40:55.754219 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.754198 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 19 14:40:55.754590 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.754415 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r"] Jul 19 14:40:55.754590 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.754502 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 19 14:40:55.754590 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.754513 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f9bqm\"" Jul 19 14:40:55.754782 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.754631 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 19 14:40:55.754782 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.754775 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 19 14:40:55.754882 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.754852 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 19 14:40:55.762946 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.762926 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb"] Jul 19 14:40:55.766632 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.766610 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66bd9878b6-whfdj"] Jul 19 14:40:55.798302 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a0d9ee4-05f8-44f3-b282-f234b9bbeb13-config\") pod \"service-ca-operator-5bcbc94c78-b6xlb\" (UID: \"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.798490 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798324 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mpff\" (UniqueName: \"kubernetes.io/projected/4a0d9ee4-05f8-44f3-b282-f234b9bbeb13-kube-api-access-7mpff\") pod \"service-ca-operator-5bcbc94c78-b6xlb\" (UID: \"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.798490 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-serving-cert\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.798490 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798430 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-serving-cert\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.798490 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798468 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.798705 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798498 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-service-ca-bundle\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.798705 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a0d9ee4-05f8-44f3-b282-f234b9bbeb13-serving-cert\") pod \"service-ca-operator-5bcbc94c78-b6xlb\" (UID: \"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.798705 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798564 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:55.798705 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-tmp\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.798705 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.798705 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.799028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798737 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-snapshots\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.799028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798766 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-stats-auth\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.799028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggj5z\" (UniqueName: \"kubernetes.io/projected/b8785e97-0280-4335-a04c-823904ee8e2f-kube-api-access-ggj5z\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:55.799028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b8785e97-0280-4335-a04c-823904ee8e2f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:55.799028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkr7t\" (UniqueName: \"kubernetes.io/projected/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-kube-api-access-qkr7t\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.799028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798932 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qfc2\" (UniqueName: \"kubernetes.io/projected/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-kube-api-access-4qfc2\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.799028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.798965 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8cv\" (UniqueName: \"kubernetes.io/projected/3d8bb224-2cdd-46bc-baaa-425a78e84984-kube-api-access-lm8cv\") pod \"network-check-source-7bc5b6bf85-ds74r\" (UID: \"3d8bb224-2cdd-46bc-baaa-425a78e84984\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r" Jul 19 14:40:55.799028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.799007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-config\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.799412 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.799041 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-trusted-ca\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.799412 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.799079 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-default-certificate\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.799412 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.799190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv9zl\" (UniqueName: \"kubernetes.io/projected/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-kube-api-access-tv9zl\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.799630 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.799604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-tmp\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.800206 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.800182 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b8785e97-0280-4335-a04c-823904ee8e2f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:55.800322 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.800203 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-service-ca-bundle\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.800427 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.800201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-snapshots\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.800486 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.800277 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.800486 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:55.800290 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:40:55.800600 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:55.800520 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls podName:b8785e97-0280-4335-a04c-823904ee8e2f nodeName:}" failed. No retries permitted until 2026-07-19 14:40:56.300496945 +0000 UTC m=+268.722721271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8gkjf" (UID: "b8785e97-0280-4335-a04c-823904ee8e2f") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:40:55.800665 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.800318 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-trusted-ca\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.800968 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.800945 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-config\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.801970 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.801952 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-serving-cert\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.802710 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.802689 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-serving-cert\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.808920 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.808895 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkr7t\" (UniqueName: \"kubernetes.io/projected/69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8-kube-api-access-qkr7t\") pod \"console-operator-565b7b569d-74gwb\" (UID: \"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8\") " pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:55.809789 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.809763 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggj5z\" (UniqueName: \"kubernetes.io/projected/b8785e97-0280-4335-a04c-823904ee8e2f-kube-api-access-ggj5z\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:55.810427 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.810404 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv9zl\" (UniqueName: \"kubernetes.io/projected/65b96bc2-0209-4d7d-aa1c-a2348b7b14b9-kube-api-access-tv9zl\") pod \"insights-operator-555d9d7885-sjhrl\" (UID: \"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9\") " pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.901666 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.901573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a0d9ee4-05f8-44f3-b282-f234b9bbeb13-serving-cert\") pod \"service-ca-operator-5bcbc94c78-b6xlb\" (UID: \"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.901666 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.901628 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.901666 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.901655 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.902062 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.901693 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-stats-auth\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.902062 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:55.901776 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:40:55.902062 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:55.901839 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:40:56.401823494 +0000 UTC m=+268.824047804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : secret "router-metrics-certs-default" not found Jul 19 14:40:55.902062 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.901857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qfc2\" (UniqueName: \"kubernetes.io/projected/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-kube-api-access-4qfc2\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.902062 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.901904 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8cv\" (UniqueName: \"kubernetes.io/projected/3d8bb224-2cdd-46bc-baaa-425a78e84984-kube-api-access-lm8cv\") pod \"network-check-source-7bc5b6bf85-ds74r\" (UID: \"3d8bb224-2cdd-46bc-baaa-425a78e84984\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r" Jul 19 14:40:55.902062 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:55.901954 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:40:56.401928507 +0000 UTC m=+268.824152822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : configmap references non-existent config key: service-ca.crt Jul 19 14:40:55.902062 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.901987 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-default-certificate\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.902062 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.902027 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a0d9ee4-05f8-44f3-b282-f234b9bbeb13-config\") pod \"service-ca-operator-5bcbc94c78-b6xlb\" (UID: \"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.902444 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.902071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mpff\" (UniqueName: \"kubernetes.io/projected/4a0d9ee4-05f8-44f3-b282-f234b9bbeb13-kube-api-access-7mpff\") pod \"service-ca-operator-5bcbc94c78-b6xlb\" (UID: \"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.902633 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.902610 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a0d9ee4-05f8-44f3-b282-f234b9bbeb13-config\") pod \"service-ca-operator-5bcbc94c78-b6xlb\" (UID: \"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.904133 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.904111 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a0d9ee4-05f8-44f3-b282-f234b9bbeb13-serving-cert\") pod \"service-ca-operator-5bcbc94c78-b6xlb\" (UID: \"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.904215 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.904185 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-default-certificate\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.904252 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.904215 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-stats-auth\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.911706 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.911683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qfc2\" (UniqueName: \"kubernetes.io/projected/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-kube-api-access-4qfc2\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:55.911948 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.911931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mpff\" (UniqueName: \"kubernetes.io/projected/4a0d9ee4-05f8-44f3-b282-f234b9bbeb13-kube-api-access-7mpff\") pod \"service-ca-operator-5bcbc94c78-b6xlb\" (UID: \"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:55.912377 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.912358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8cv\" (UniqueName: \"kubernetes.io/projected/3d8bb224-2cdd-46bc-baaa-425a78e84984-kube-api-access-lm8cv\") pod \"network-check-source-7bc5b6bf85-ds74r\" (UID: \"3d8bb224-2cdd-46bc-baaa-425a78e84984\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r" Jul 19 14:40:55.962695 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.962661 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-sjhrl" Jul 19 14:40:55.969352 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:55.969331 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:40:56.053167 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.053139 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r" Jul 19 14:40:56.061024 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.061003 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" Jul 19 14:40:56.107543 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.106639 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-sjhrl"] Jul 19 14:40:56.110514 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:40:56.110480 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b96bc2_0209_4d7d_aa1c_a2348b7b14b9.slice/crio-a25b81e72a6b6970b7819ca8e2ac3dadb44ef05d5766dd6a96b6119c29077714 WatchSource:0}: Error finding container a25b81e72a6b6970b7819ca8e2ac3dadb44ef05d5766dd6a96b6119c29077714: Status 404 returned error can't find the container with id a25b81e72a6b6970b7819ca8e2ac3dadb44ef05d5766dd6a96b6119c29077714 Jul 19 14:40:56.128751 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.128160 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-74gwb"] Jul 19 14:40:56.131934 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:40:56.131900 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e1fb02_21d0_4cb1_b7fc_f5fb1ba93ac8.slice/crio-b029463f5d2019197f98f375682dc86a3facde85c1aa92ffed6f4609cdb115aa WatchSource:0}: Error finding container b029463f5d2019197f98f375682dc86a3facde85c1aa92ffed6f4609cdb115aa: Status 404 returned error can't find the container with id b029463f5d2019197f98f375682dc86a3facde85c1aa92ffed6f4609cdb115aa Jul 19 14:40:56.189538 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.189347 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r"] Jul 19 14:40:56.191759 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:40:56.191737 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8bb224_2cdd_46bc_baaa_425a78e84984.slice/crio-b5f0df51b5717662a525fde1c9f7ef4ef6ef919abe1bf3201f7100387a094e9a WatchSource:0}: Error finding container b5f0df51b5717662a525fde1c9f7ef4ef6ef919abe1bf3201f7100387a094e9a: Status 404 returned error can't find the container with id b5f0df51b5717662a525fde1c9f7ef4ef6ef919abe1bf3201f7100387a094e9a Jul 19 14:40:56.204421 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.204398 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb"] Jul 19 14:40:56.207489 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:40:56.207461 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0d9ee4_05f8_44f3_b282_f234b9bbeb13.slice/crio-f42283aa8fefe7edf14306188e2904f5b2a5e244831c72e98c4291ecbf4afc7d WatchSource:0}: Error finding container f42283aa8fefe7edf14306188e2904f5b2a5e244831c72e98c4291ecbf4afc7d: Status 404 returned error can't find the container with id f42283aa8fefe7edf14306188e2904f5b2a5e244831c72e98c4291ecbf4afc7d Jul 19 14:40:56.305127 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.305086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:56.305274 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:56.305255 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:40:56.305343 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:56.305332 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls podName:b8785e97-0280-4335-a04c-823904ee8e2f nodeName:}" failed. No retries permitted until 2026-07-19 14:40:57.305312049 +0000 UTC m=+269.727536361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8gkjf" (UID: "b8785e97-0280-4335-a04c-823904ee8e2f") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:40:56.405817 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.405791 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:56.405961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.405831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:56.405961 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:56.405948 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:40:56.406060 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:56.405979 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:40:57.405966517 +0000 UTC m=+269.828190827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : configmap references non-existent config key: service-ca.crt Jul 19 14:40:56.406060 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:56.405995 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:40:57.40598955 +0000 UTC m=+269.828213860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : secret "router-metrics-certs-default" not found Jul 19 14:40:56.812129 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.812045 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" event={"ID":"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13","Type":"ContainerStarted","Data":"f42283aa8fefe7edf14306188e2904f5b2a5e244831c72e98c4291ecbf4afc7d"} Jul 19 14:40:56.816033 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.815605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r" event={"ID":"3d8bb224-2cdd-46bc-baaa-425a78e84984","Type":"ContainerStarted","Data":"cf53e01446feed8ad13ac3301aded401ce3439620082b14c4584a9c07a4799ea"} Jul 19 14:40:56.816033 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.815653 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r" event={"ID":"3d8bb224-2cdd-46bc-baaa-425a78e84984","Type":"ContainerStarted","Data":"b5f0df51b5717662a525fde1c9f7ef4ef6ef919abe1bf3201f7100387a094e9a"} Jul 19 14:40:56.819355 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.819289 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" event={"ID":"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8","Type":"ContainerStarted","Data":"b029463f5d2019197f98f375682dc86a3facde85c1aa92ffed6f4609cdb115aa"} Jul 19 14:40:56.822382 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.822356 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sjhrl" event={"ID":"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9","Type":"ContainerStarted","Data":"a25b81e72a6b6970b7819ca8e2ac3dadb44ef05d5766dd6a96b6119c29077714"} Jul 19 14:40:56.832933 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:56.832845 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ds74r" podStartSLOduration=1.83282898 podStartE2EDuration="1.83282898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:40:56.832048341 +0000 UTC m=+269.254272674" watchObservedRunningTime="2026-07-19 14:40:56.83282898 +0000 UTC m=+269.255053312" Jul 19 14:40:57.317767 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:57.313753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:57.317767 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:57.313962 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:40:57.317767 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:57.314063 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls podName:b8785e97-0280-4335-a04c-823904ee8e2f nodeName:}" failed. No retries permitted until 2026-07-19 14:40:59.314025225 +0000 UTC m=+271.736249539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8gkjf" (UID: "b8785e97-0280-4335-a04c-823904ee8e2f") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:40:57.415338 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:57.415299 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:57.415550 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:57.415358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:57.415624 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:57.415550 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:40:59.415532613 +0000 UTC m=+271.837756930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : configmap references non-existent config key: service-ca.crt Jul 19 14:40:57.416023 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:57.415945 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:40:57.416023 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:57.415992 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:40:59.415979383 +0000 UTC m=+271.838203700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : secret "router-metrics-certs-default" not found Jul 19 14:40:59.331702 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.331676 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:40:59.332176 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:59.331845 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:40:59.332176 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:59.331930 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls podName:b8785e97-0280-4335-a04c-823904ee8e2f nodeName:}" failed. No retries permitted until 2026-07-19 14:41:03.33190971 +0000 UTC m=+275.754134019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8gkjf" (UID: "b8785e97-0280-4335-a04c-823904ee8e2f") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:40:59.432760 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.432712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:59.432949 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.432774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:40:59.432949 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:59.432840 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:40:59.432949 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:59.432909 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:41:03.432891449 +0000 UTC m=+275.855115776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : secret "router-metrics-certs-default" not found Jul 19 14:40:59.433128 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:40:59.432985 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:41:03.432970046 +0000 UTC m=+275.855194368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : configmap references non-existent config key: service-ca.crt Jul 19 14:40:59.833203 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.833168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" event={"ID":"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13","Type":"ContainerStarted","Data":"3bfaecdb948d7ad1b7e1196dbd5fd3e287571677d161dbfe3225c711c10d8b5d"} Jul 19 14:40:59.834807 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.834782 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/0.log" Jul 19 14:40:59.834946 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.834822 2559 generic.go:356] "Generic (PLEG): container finished" podID="69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8" containerID="23e1c3c0b9754b3030ab4ecffc4ce6a89199446f34d55a8478e79d06d767edd7" exitCode=255 Jul 19 14:40:59.834946 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.834911 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" event={"ID":"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8","Type":"ContainerDied","Data":"23e1c3c0b9754b3030ab4ecffc4ce6a89199446f34d55a8478e79d06d767edd7"} Jul 19 14:40:59.835149 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.835134 2559 scope.go:117] "RemoveContainer" containerID="23e1c3c0b9754b3030ab4ecffc4ce6a89199446f34d55a8478e79d06d767edd7" Jul 19 14:40:59.836401 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.836378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sjhrl" event={"ID":"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9","Type":"ContainerStarted","Data":"20478defc216d05e9bb208525106dc346e79e81338eb983e3e54d401f27ffea3"} Jul 19 14:40:59.875390 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.875178 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" podStartSLOduration=1.746057869 podStartE2EDuration="4.875159604s" totalImagesPullingTime="3.129101735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:40:59.853056622 +0000 UTC m=+272.275280966" watchObservedRunningTime="2026-07-19 14:40:59.875159604 +0000 UTC m=+272.297383938" Jul 19 14:40:59.891284 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:40:59.891223 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-sjhrl" podStartSLOduration=1.667151373 podStartE2EDuration="4.891209753s" totalImagesPullingTime="3.22405838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:40:59.889724028 +0000 UTC m=+272.311948356" watchObservedRunningTime="2026-07-19 14:40:59.891209753 +0000 UTC m=+272.313434086" Jul 19 14:41:00.178892 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.178777 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v69md"] Jul 19 14:41:00.182423 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.182396 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.184801 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.184768 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 19 14:41:00.184939 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.184775 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 19 14:41:00.184939 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.184816 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4qjwr\"" Jul 19 14:41:00.193043 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.193005 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v69md"] Jul 19 14:41:00.240561 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.240521 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6278e4df-2a03-458e-bcac-330d3ab063cc-data-volume\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.240711 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.240661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.240750 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.240709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.240750 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.240738 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6278e4df-2a03-458e-bcac-330d3ab063cc-crio-socket\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.240854 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.240764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgxcz\" (UniqueName: \"kubernetes.io/projected/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-api-access-tgxcz\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.341417 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.341378 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6278e4df-2a03-458e-bcac-330d3ab063cc-data-volume\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.341467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.341494 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.341511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6278e4df-2a03-458e-bcac-330d3ab063cc-crio-socket\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.341528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgxcz\" (UniqueName: \"kubernetes.io/projected/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-api-access-tgxcz\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:00.341619 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.341633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6278e4df-2a03-458e-bcac-330d3ab063cc-crio-socket\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:00.341659 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:00.341694 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:00.841674256 +0000 UTC m=+273.263898584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : secret "insights-runtime-extractor-tls" not found Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:00.341723 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:00.841703818 +0000 UTC m=+273.263928129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : configmap "kube-rbac-proxy" not found Jul 19 14:41:00.341846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.341740 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6278e4df-2a03-458e-bcac-330d3ab063cc-data-volume\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.349806 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.349785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgxcz\" (UniqueName: \"kubernetes.io/projected/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-api-access-tgxcz\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.841215 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.841186 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:41:00.841622 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.841606 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/0.log" Jul 19 14:41:00.841687 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.841640 2559 generic.go:356] "Generic (PLEG): container finished" podID="69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8" containerID="27d927588c97a83af5f8e7ba24bb146690e2e1eb7e8f4b98b7f1c9ca1423d9a1" exitCode=255 Jul 19 14:41:00.841754 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.841730 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" event={"ID":"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8","Type":"ContainerDied","Data":"27d927588c97a83af5f8e7ba24bb146690e2e1eb7e8f4b98b7f1c9ca1423d9a1"} Jul 19 14:41:00.842658 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.842641 2559 scope.go:117] "RemoveContainer" containerID="23e1c3c0b9754b3030ab4ecffc4ce6a89199446f34d55a8478e79d06d767edd7" Jul 19 14:41:00.842756 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.842739 2559 scope.go:117] "RemoveContainer" containerID="27d927588c97a83af5f8e7ba24bb146690e2e1eb7e8f4b98b7f1c9ca1423d9a1" Jul 19 14:41:00.842931 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:00.842904 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-74gwb_openshift-console-operator(69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8)\"" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" podUID="69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8" Jul 19 14:41:00.844745 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.844725 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.844821 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:00.844761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:00.844855 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:00.844842 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:41:00.844903 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:00.844859 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:41:00.844943 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:00.844912 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:01.844897717 +0000 UTC m=+274.267122027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : configmap "kube-rbac-proxy" not found Jul 19 14:41:00.844943 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:00.844925 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:01.84492013 +0000 UTC m=+274.267144439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : secret "insights-runtime-extractor-tls" not found Jul 19 14:41:01.849647 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:01.849618 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:41:01.850055 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:01.849933 2559 scope.go:117] "RemoveContainer" containerID="27d927588c97a83af5f8e7ba24bb146690e2e1eb7e8f4b98b7f1c9ca1423d9a1" Jul 19 14:41:01.850125 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:01.850105 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-74gwb_openshift-console-operator(69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8)\"" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" podUID="69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8" Jul 19 14:41:01.853238 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:01.853221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:01.853282 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:01.853265 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:01.853355 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:01.853342 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:41:01.853391 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:01.853342 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:41:01.853423 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:01.853391 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:03.853376345 +0000 UTC m=+276.275600654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : secret "insights-runtime-extractor-tls" not found Jul 19 14:41:01.853468 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:01.853459 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:03.853441814 +0000 UTC m=+276.275666128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : configmap "kube-rbac-proxy" not found Jul 19 14:41:02.708015 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.707980 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-mmdqw"] Jul 19 14:41:02.712274 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.712252 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:02.714840 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.714815 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 19 14:41:02.714968 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.714841 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 19 14:41:02.714968 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.714846 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 19 14:41:02.714968 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.714916 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 19 14:41:02.715740 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.715727 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-brpfb\"" Jul 19 14:41:02.720227 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.720209 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-mmdqw"] Jul 19 14:41:02.761811 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.761776 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d03e4f1-d86a-4f49-8190-2594bc9bfb9b-signing-key\") pod \"service-ca-58598695f7-mmdqw\" (UID: \"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b\") " pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:02.761811 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.761815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhqw\" (UniqueName: \"kubernetes.io/projected/7d03e4f1-d86a-4f49-8190-2594bc9bfb9b-kube-api-access-bjhqw\") pod \"service-ca-58598695f7-mmdqw\" (UID: \"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b\") " pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:02.762019 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.761839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d03e4f1-d86a-4f49-8190-2594bc9bfb9b-signing-cabundle\") pod \"service-ca-58598695f7-mmdqw\" (UID: \"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b\") " pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:02.863179 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.863149 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d03e4f1-d86a-4f49-8190-2594bc9bfb9b-signing-key\") pod \"service-ca-58598695f7-mmdqw\" (UID: \"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b\") " pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:02.863560 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.863188 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhqw\" (UniqueName: \"kubernetes.io/projected/7d03e4f1-d86a-4f49-8190-2594bc9bfb9b-kube-api-access-bjhqw\") pod \"service-ca-58598695f7-mmdqw\" (UID: \"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b\") " pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:02.863560 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.863211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d03e4f1-d86a-4f49-8190-2594bc9bfb9b-signing-cabundle\") pod \"service-ca-58598695f7-mmdqw\" (UID: \"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b\") " pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:02.863918 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.863899 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d03e4f1-d86a-4f49-8190-2594bc9bfb9b-signing-cabundle\") pod \"service-ca-58598695f7-mmdqw\" (UID: \"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b\") " pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:02.865767 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.865748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d03e4f1-d86a-4f49-8190-2594bc9bfb9b-signing-key\") pod \"service-ca-58598695f7-mmdqw\" (UID: \"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b\") " pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:02.871322 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:02.871303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhqw\" (UniqueName: \"kubernetes.io/projected/7d03e4f1-d86a-4f49-8190-2594bc9bfb9b-kube-api-access-bjhqw\") pod \"service-ca-58598695f7-mmdqw\" (UID: \"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b\") " pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:03.022059 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.021925 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-mmdqw" Jul 19 14:41:03.070230 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.070205 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7rprf_e0307f2d-fa2f-42dd-984e-148f44510d07/dns-node-resolver/0.log" Jul 19 14:41:03.147347 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.147259 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-mmdqw"] Jul 19 14:41:03.149580 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:03.149549 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d03e4f1_d86a_4f49_8190_2594bc9bfb9b.slice/crio-8800efd7f06e0600be65a91d1f9b21895784c6969f6fef6779620cad9bb8d371 WatchSource:0}: Error finding container 8800efd7f06e0600be65a91d1f9b21895784c6969f6fef6779620cad9bb8d371: Status 404 returned error can't find the container with id 8800efd7f06e0600be65a91d1f9b21895784c6969f6fef6779620cad9bb8d371 Jul 19 14:41:03.367524 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.367496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:41:03.367665 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:03.367646 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:41:03.367717 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:03.367708 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls podName:b8785e97-0280-4335-a04c-823904ee8e2f nodeName:}" failed. No retries permitted until 2026-07-19 14:41:11.367693813 +0000 UTC m=+283.789918122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8gkjf" (UID: "b8785e97-0280-4335-a04c-823904ee8e2f") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:41:03.468721 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.468692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:03.468897 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.468730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:03.468897 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:03.468844 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:41:11.468831314 +0000 UTC m=+283.891055624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : configmap references non-existent config key: service-ca.crt Jul 19 14:41:03.468897 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:03.468846 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:41:03.469032 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:03.468922 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs podName:05139392-8ed3-48d4-91b1-f6a56d0b4e5d nodeName:}" failed. No retries permitted until 2026-07-19 14:41:11.468907112 +0000 UTC m=+283.891131441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs") pod "router-default-66bd9878b6-whfdj" (UID: "05139392-8ed3-48d4-91b1-f6a56d0b4e5d") : secret "router-metrics-certs-default" not found Jul 19 14:41:03.856716 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.856679 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-mmdqw" event={"ID":"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b","Type":"ContainerStarted","Data":"cc7fb89247305b6e74da645244348e9c7a9b56c436149379ce3a07f80d243002"} Jul 19 14:41:03.856898 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.856736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-mmdqw" event={"ID":"7d03e4f1-d86a-4f49-8190-2594bc9bfb9b","Type":"ContainerStarted","Data":"8800efd7f06e0600be65a91d1f9b21895784c6969f6fef6779620cad9bb8d371"} Jul 19 14:41:03.872739 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.872714 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:03.873106 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.872764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:03.873106 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:03.872858 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:41:03.873106 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:03.872915 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:41:03.873106 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:03.872927 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:07.872913257 +0000 UTC m=+280.295137567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : secret "insights-runtime-extractor-tls" not found Jul 19 14:41:03.873106 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:03.872957 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:07.8729455 +0000 UTC m=+280.295169809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : configmap "kube-rbac-proxy" not found Jul 19 14:41:03.874918 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:03.874857 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-mmdqw" podStartSLOduration=1.8748434299999999 podStartE2EDuration="1.87484343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:03.873548284 +0000 UTC m=+276.295772613" watchObservedRunningTime="2026-07-19 14:41:03.87484343 +0000 UTC m=+276.297067761" Jul 19 14:41:04.301627 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:04.301553 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lgdc4_7b223bb6-1dec-46f4-a6a8-7204b718d631/node-ca/0.log" Jul 19 14:41:04.670548 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:04.670513 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tkvk_017827df-343d-42db-beec-ce9b21b305f5/serve-healthcheck-canary/0.log" Jul 19 14:41:05.969975 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:05.969857 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:41:05.970459 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:05.969980 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:41:05.970459 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:05.970425 2559 scope.go:117] "RemoveContainer" containerID="27d927588c97a83af5f8e7ba24bb146690e2e1eb7e8f4b98b7f1c9ca1423d9a1" Jul 19 14:41:05.970653 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:05.970628 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-74gwb_openshift-console-operator(69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8)\"" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" podUID="69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8" Jul 19 14:41:07.907414 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:07.907374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:07.907791 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:07.907434 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:07.907791 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:07.907511 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:41:07.907791 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:07.907518 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:41:07.907791 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:07.907570 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:15.907554296 +0000 UTC m=+288.329778605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : configmap "kube-rbac-proxy" not found Jul 19 14:41:07.907791 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:07.907595 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls podName:6278e4df-2a03-458e-bcac-330d3ab063cc nodeName:}" failed. No retries permitted until 2026-07-19 14:41:15.907580314 +0000 UTC m=+288.329804638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v69md" (UID: "6278e4df-2a03-458e-bcac-330d3ab063cc") : secret "insights-runtime-extractor-tls" not found Jul 19 14:41:08.549842 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:08.549797 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" podUID="68cdeba4-359c-4409-8c6d-8297f763352b" Jul 19 14:41:08.549842 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:08.549797 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" podUID="30c83ee5-7890-4240-957c-49de880b15a4" Jul 19 14:41:08.549842 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:08.549797 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bgdc6" podUID="3664d65b-c2f1-4254-a294-9f4123067a5b" Jul 19 14:41:08.876964 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:08.876930 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bgdc6" Jul 19 14:41:08.876964 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:08.876966 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:41:08.877197 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:08.876971 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:41:11.434789 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.434748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:41:11.437072 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.437049 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8785e97-0280-4335-a04c-823904ee8e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8gkjf\" (UID: \"b8785e97-0280-4335-a04c-823904ee8e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:41:11.536018 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.535980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:11.536018 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.536025 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:11.536570 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.536541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-service-ca-bundle\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:11.538295 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.538270 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05139392-8ed3-48d4-91b1-f6a56d0b4e5d-metrics-certs\") pod \"router-default-66bd9878b6-whfdj\" (UID: \"05139392-8ed3-48d4-91b1-f6a56d0b4e5d\") " pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:11.575756 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.575730 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" Jul 19 14:41:11.666884 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.666614 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:11.702076 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.701973 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf"] Jul 19 14:41:11.706019 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:11.705983 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8785e97_0280_4335_a04c_823904ee8e2f.slice/crio-60288fbeb9b0a3a40c7834ca40b0e560ffd2882d7fdbacd09e0dac2aef64feac WatchSource:0}: Error finding container 60288fbeb9b0a3a40c7834ca40b0e560ffd2882d7fdbacd09e0dac2aef64feac: Status 404 returned error can't find the container with id 60288fbeb9b0a3a40c7834ca40b0e560ffd2882d7fdbacd09e0dac2aef64feac Jul 19 14:41:11.795815 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.795786 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66bd9878b6-whfdj"] Jul 19 14:41:11.798916 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:11.798858 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05139392_8ed3_48d4_91b1_f6a56d0b4e5d.slice/crio-2d63624f8aab5cc58de960404c84ef7ce2517ce5d9601f78578f30e692fac818 WatchSource:0}: Error finding container 2d63624f8aab5cc58de960404c84ef7ce2517ce5d9601f78578f30e692fac818: Status 404 returned error can't find the container with id 2d63624f8aab5cc58de960404c84ef7ce2517ce5d9601f78578f30e692fac818 Jul 19 14:41:11.892814 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.892781 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66bd9878b6-whfdj" event={"ID":"05139392-8ed3-48d4-91b1-f6a56d0b4e5d","Type":"ContainerStarted","Data":"11d1ee92a7b2c8e4db9928aa87f602901891154b8cd6dbee91e92e80f935cda5"} Jul 19 14:41:11.892952 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.892838 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66bd9878b6-whfdj" event={"ID":"05139392-8ed3-48d4-91b1-f6a56d0b4e5d","Type":"ContainerStarted","Data":"2d63624f8aab5cc58de960404c84ef7ce2517ce5d9601f78578f30e692fac818"} Jul 19 14:41:11.893973 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.893948 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" event={"ID":"b8785e97-0280-4335-a04c-823904ee8e2f","Type":"ContainerStarted","Data":"60288fbeb9b0a3a40c7834ca40b0e560ffd2882d7fdbacd09e0dac2aef64feac"} Jul 19 14:41:11.911876 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.911820 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66bd9878b6-whfdj" podStartSLOduration=16.911804284 podStartE2EDuration="16.911804284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:11.91113168 +0000 UTC m=+284.333356012" watchObservedRunningTime="2026-07-19 14:41:11.911804284 +0000 UTC m=+284.334028616" Jul 19 14:41:11.940139 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.940112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:41:11.940412 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.940390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:41:11.942662 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.942635 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"image-registry-666d9b4699-8pbzg\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:41:11.942819 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:11.942798 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3664d65b-c2f1-4254-a294-9f4123067a5b-metrics-tls\") pod \"dns-default-bgdc6\" (UID: \"3664d65b-c2f1-4254-a294-9f4123067a5b\") " pod="openshift-dns/dns-default-bgdc6" Jul 19 14:41:12.041234 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.041153 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:41:12.041383 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:12.041308 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:41:12.041440 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:12.041400 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert podName:30c83ee5-7890-4240-957c-49de880b15a4 nodeName:}" failed. No retries permitted until 2026-07-19 14:43:14.041378938 +0000 UTC m=+406.463603263 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2dcp2" (UID: "30c83ee5-7890-4240-957c-49de880b15a4") : secret "networking-console-plugin-cert" not found Jul 19 14:41:12.179562 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.179533 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8vqq7\"" Jul 19 14:41:12.179562 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.179532 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rz27h\"" Jul 19 14:41:12.188111 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.188082 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bgdc6" Jul 19 14:41:12.188218 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.188172 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:41:12.347665 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.347635 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bgdc6"] Jul 19 14:41:12.352097 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:12.352061 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3664d65b_c2f1_4254_a294_9f4123067a5b.slice/crio-68882227a180c3c91c00d6afdd1b5fbc622bc39f9cfd2a2f4c2814985a0576d0 WatchSource:0}: Error finding container 68882227a180c3c91c00d6afdd1b5fbc622bc39f9cfd2a2f4c2814985a0576d0: Status 404 returned error can't find the container with id 68882227a180c3c91c00d6afdd1b5fbc622bc39f9cfd2a2f4c2814985a0576d0 Jul 19 14:41:12.361737 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.361712 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666d9b4699-8pbzg"] Jul 19 14:41:12.363712 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:12.363682 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cdeba4_359c_4409_8c6d_8297f763352b.slice/crio-d4af97d41c6161baa3e6120abaf188b9b2ecab45d4e6c85753fe764d7e464d59 WatchSource:0}: Error finding container d4af97d41c6161baa3e6120abaf188b9b2ecab45d4e6c85753fe764d7e464d59: Status 404 returned error can't find the container with id d4af97d41c6161baa3e6120abaf188b9b2ecab45d4e6c85753fe764d7e464d59 Jul 19 14:41:12.667256 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.667170 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:12.670615 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.670405 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:12.899580 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.899221 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" event={"ID":"68cdeba4-359c-4409-8c6d-8297f763352b","Type":"ContainerStarted","Data":"c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a"} Jul 19 14:41:12.899580 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.899307 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" event={"ID":"68cdeba4-359c-4409-8c6d-8297f763352b","Type":"ContainerStarted","Data":"d4af97d41c6161baa3e6120abaf188b9b2ecab45d4e6c85753fe764d7e464d59"} Jul 19 14:41:12.899580 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.899373 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:41:12.900752 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.900725 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bgdc6" event={"ID":"3664d65b-c2f1-4254-a294-9f4123067a5b","Type":"ContainerStarted","Data":"68882227a180c3c91c00d6afdd1b5fbc622bc39f9cfd2a2f4c2814985a0576d0"} Jul 19 14:41:12.900916 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.900897 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:12.902475 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.902451 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66bd9878b6-whfdj" Jul 19 14:41:12.917581 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:12.917497 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" podStartSLOduration=284.917483909 podStartE2EDuration="4m44.917483909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:12.917169229 +0000 UTC m=+285.339393563" watchObservedRunningTime="2026-07-19 14:41:12.917483909 +0000 UTC m=+285.339708219" Jul 19 14:41:13.910843 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:13.910278 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" event={"ID":"b8785e97-0280-4335-a04c-823904ee8e2f","Type":"ContainerStarted","Data":"878d08f3c39ac8c2116263f75a0ef2318fbd58f9afe6823d21666b77becfa282"} Jul 19 14:41:13.925215 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:13.925160 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8gkjf" podStartSLOduration=16.858113101 podStartE2EDuration="18.925139848s" totalImagesPullingTime="2.067026747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:13.924628975 +0000 UTC m=+286.346853307" watchObservedRunningTime="2026-07-19 14:41:13.925139848 +0000 UTC m=+286.347364180" Jul 19 14:41:14.912565 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:14.912526 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bgdc6" event={"ID":"3664d65b-c2f1-4254-a294-9f4123067a5b","Type":"ContainerStarted","Data":"7fbdecfb44967fb7ead834d6b3d04b5da74076a3ba09e46a22777c1804b2d21e"} Jul 19 14:41:14.912925 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:14.912606 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bgdc6" event={"ID":"3664d65b-c2f1-4254-a294-9f4123067a5b","Type":"ContainerStarted","Data":"3a2beab1bff4860881ff6460350c742f47040d434b4304e525e00d7efd609d64"} Jul 19 14:41:14.929268 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:14.929225 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bgdc6" podStartSLOduration=251.80556443 podStartE2EDuration="4m13.929210829s" totalImagesPullingTime="2.123646399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:14.928268796 +0000 UTC m=+287.350493127" watchObservedRunningTime="2026-07-19 14:41:14.929210829 +0000 UTC m=+287.351435159" Jul 19 14:41:15.915979 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:15.915954 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bgdc6" Jul 19 14:41:15.976131 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:15.976095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:15.976283 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:15.976148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:15.976768 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:15.976742 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6278e4df-2a03-458e-bcac-330d3ab063cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:15.978343 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:15.978317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6278e4df-2a03-458e-bcac-330d3ab063cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v69md\" (UID: \"6278e4df-2a03-458e-bcac-330d3ab063cc\") " pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:16.091754 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:16.091718 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v69md" Jul 19 14:41:16.214268 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:16.214232 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v69md"] Jul 19 14:41:16.217233 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:16.217211 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6278e4df_2a03_458e_bcac_330d3ab063cc.slice/crio-8caa0f6790658fa7bcb61663c677718001dc663b7be1fa010732d319b51e8df2 WatchSource:0}: Error finding container 8caa0f6790658fa7bcb61663c677718001dc663b7be1fa010732d319b51e8df2: Status 404 returned error can't find the container with id 8caa0f6790658fa7bcb61663c677718001dc663b7be1fa010732d319b51e8df2 Jul 19 14:41:16.921436 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:16.921399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v69md" event={"ID":"6278e4df-2a03-458e-bcac-330d3ab063cc","Type":"ContainerStarted","Data":"8caa0f6790658fa7bcb61663c677718001dc663b7be1fa010732d319b51e8df2"} Jul 19 14:41:17.101627 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:17.101593 2559 scope.go:117] "RemoveContainer" containerID="27d927588c97a83af5f8e7ba24bb146690e2e1eb7e8f4b98b7f1c9ca1423d9a1" Jul 19 14:41:17.927303 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:17.927272 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:41:17.927764 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:17.927369 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" event={"ID":"69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8","Type":"ContainerStarted","Data":"3b3658fbbc1f4ac29360ceedce88e501f2014549d6c23012f2e88b6a85693a62"} Jul 19 14:41:17.927821 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:17.927799 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:41:17.944501 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:17.944446 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" podStartSLOduration=19.748301586 podStartE2EDuration="22.944430572s" totalImagesPullingTime="3.196128986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:17.942732002 +0000 UTC m=+290.364956347" watchObservedRunningTime="2026-07-19 14:41:17.944430572 +0000 UTC m=+290.366654944" Jul 19 14:41:18.319765 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:18.319689 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-74gwb" Jul 19 14:41:21.945079 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:21.945033 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v69md" event={"ID":"6278e4df-2a03-458e-bcac-330d3ab063cc","Type":"ContainerStarted","Data":"4c7ebeb98a808adb9449c48fdb2586cebbaa85215d23c5a4befe9d186b84cced"} Jul 19 14:41:24.102966 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:41:24.102919 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zw8ds" podUID="82fc003a-678c-4e3d-90ab-0c64f693f817" Jul 19 14:41:24.957726 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:24.957692 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v69md" event={"ID":"6278e4df-2a03-458e-bcac-330d3ab063cc","Type":"ContainerStarted","Data":"fa41d998a4eab0b895268d17f58f8a7c614b8d04f57fb76afa6c169cc3510d90"} Jul 19 14:41:25.924148 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:25.924114 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bgdc6" Jul 19 14:41:27.408202 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.408172 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55bb554f8b-qzhb7"] Jul 19 14:41:27.427670 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.427603 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bb554f8b-qzhb7"] Jul 19 14:41:27.427845 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.427737 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.430605 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.430571 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 19 14:41:27.430721 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.430685 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 19 14:41:27.430721 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.430686 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 19 14:41:27.431494 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.431468 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 19 14:41:27.431603 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.431495 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 19 14:41:27.431603 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.431506 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 19 14:41:27.431857 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.431838 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lzfm7\"" Jul 19 14:41:27.431952 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.431908 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 19 14:41:27.578184 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.578153 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-config\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.578354 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.578200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-service-ca\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.578354 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.578294 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4x72\" (UniqueName: \"kubernetes.io/projected/b09ee3a0-7b5d-4106-9d17-207120cc0aef-kube-api-access-h4x72\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.578354 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.578323 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-serving-cert\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.578497 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.578396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-oauth-config\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.578497 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.578449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-oauth-serving-cert\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.679677 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.679599 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-config\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.679677 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.679646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-service-ca\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.679896 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.679698 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4x72\" (UniqueName: \"kubernetes.io/projected/b09ee3a0-7b5d-4106-9d17-207120cc0aef-kube-api-access-h4x72\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.679896 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.679786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-serving-cert\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.679896 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.679845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-oauth-config\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.679994 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.679901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-oauth-serving-cert\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.680566 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.680541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-service-ca\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.680688 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.680599 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-config\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.680688 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.680617 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-oauth-serving-cert\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.682683 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.682658 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-serving-cert\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.682801 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.682688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-oauth-config\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.687243 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.687220 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4x72\" (UniqueName: \"kubernetes.io/projected/b09ee3a0-7b5d-4106-9d17-207120cc0aef-kube-api-access-h4x72\") pod \"console-55bb554f8b-qzhb7\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:27.738061 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:27.738018 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:28.012593 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:28.012573 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:41:28.012924 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:28.012903 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:41:28.022300 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:28.021988 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bb554f8b-qzhb7"] Jul 19 14:41:28.025195 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:28.025157 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09ee3a0_7b5d_4106_9d17_207120cc0aef.slice/crio-a5e801424d67990bc51602ed211f1e6e1307aec7a6ef13785b080a9fd3725c47 WatchSource:0}: Error finding container a5e801424d67990bc51602ed211f1e6e1307aec7a6ef13785b080a9fd3725c47: Status 404 returned error can't find the container with id a5e801424d67990bc51602ed211f1e6e1307aec7a6ef13785b080a9fd3725c47 Jul 19 14:41:28.025458 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:28.025439 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 19 14:41:28.976646 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:28.976609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v69md" event={"ID":"6278e4df-2a03-458e-bcac-330d3ab063cc","Type":"ContainerStarted","Data":"57cfdf384ee2b86f9773a53c181c55b4fbbb88de066db5a57a49363c15db5037"} Jul 19 14:41:28.979067 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:28.977802 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bb554f8b-qzhb7" event={"ID":"b09ee3a0-7b5d-4106-9d17-207120cc0aef","Type":"ContainerStarted","Data":"a5e801424d67990bc51602ed211f1e6e1307aec7a6ef13785b080a9fd3725c47"} Jul 19 14:41:28.994560 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:28.994503 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v69md" podStartSLOduration=17.371774145 podStartE2EDuration="28.994488958s" totalImagesPullingTime="11.622714813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:41:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:28.993947078 +0000 UTC m=+301.416171409" watchObservedRunningTime="2026-07-19 14:41:28.994488958 +0000 UTC m=+301.416713307" Jul 19 14:41:29.341600 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.341568 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk"] Jul 19 14:41:29.345040 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.345018 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.348247 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.348223 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 19 14:41:29.348556 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.348525 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gxxb4\"" Jul 19 14:41:29.348665 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.348571 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 19 14:41:29.349010 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.348990 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 19 14:41:29.355464 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.355442 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk"] Jul 19 14:41:29.495131 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.495089 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.495315 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.495144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.495315 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.495176 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.495315 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.495278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8rc\" (UniqueName: \"kubernetes.io/projected/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-kube-api-access-dm8rc\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.596431 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.596353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.596431 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.596399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.596660 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.596479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.596660 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.596570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8rc\" (UniqueName: \"kubernetes.io/projected/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-kube-api-access-dm8rc\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.597299 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.597273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.599280 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.599261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.599397 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.599300 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.604652 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.604632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8rc\" (UniqueName: \"kubernetes.io/projected/c4fc61cf-4d7b-4764-84b3-f05e0dbe2402-kube-api-access-dm8rc\") pod \"prometheus-operator-5945b5cfd-bpqtk\" (UID: \"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.657545 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.657514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" Jul 19 14:41:29.796083 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.796024 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk"] Jul 19 14:41:29.798171 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:29.798136 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fc61cf_4d7b_4764_84b3_f05e0dbe2402.slice/crio-880a39628209e770085fdd0112f0f72a5a6db9abeb81b75daca99e20fff8a6c9 WatchSource:0}: Error finding container 880a39628209e770085fdd0112f0f72a5a6db9abeb81b75daca99e20fff8a6c9: Status 404 returned error can't find the container with id 880a39628209e770085fdd0112f0f72a5a6db9abeb81b75daca99e20fff8a6c9 Jul 19 14:41:29.800349 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.800332 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 14:41:29.982990 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:29.982895 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" event={"ID":"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402","Type":"ContainerStarted","Data":"880a39628209e770085fdd0112f0f72a5a6db9abeb81b75daca99e20fff8a6c9"} Jul 19 14:41:31.991480 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:31.991448 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bb554f8b-qzhb7" event={"ID":"b09ee3a0-7b5d-4106-9d17-207120cc0aef","Type":"ContainerStarted","Data":"6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d"} Jul 19 14:41:32.012148 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:32.012098 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55bb554f8b-qzhb7" podStartSLOduration=1.960659606 podStartE2EDuration="5.012080283s" totalImagesPullingTime="3.051420677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:32.010671933 +0000 UTC m=+304.432896287" watchObservedRunningTime="2026-07-19 14:41:32.012080283 +0000 UTC m=+304.434304612" Jul 19 14:41:32.193173 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:32.193146 2559 patch_prober.go:28] interesting pod/image-registry-666d9b4699-8pbzg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 19 14:41:32.193272 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:32.193198 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" podUID="68cdeba4-359c-4409-8c6d-8297f763352b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 19 14:41:32.996518 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:32.996479 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" event={"ID":"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402","Type":"ContainerStarted","Data":"04ff3a18596d60815ff988d20787a7280750a9f6c1ef21749f49505a0f1f1bdf"} Jul 19 14:41:32.996961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:32.996579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" event={"ID":"c4fc61cf-4d7b-4764-84b3-f05e0dbe2402","Type":"ContainerStarted","Data":"2bf93b6d4e557179591e2408e0d79cfe043319e665286a7b02ca5721d36008f3"} Jul 19 14:41:33.012448 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:33.012393 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bpqtk" podStartSLOduration=1.672381428 podStartE2EDuration="4.012376047s" totalImagesPullingTime="2.339994619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:33.011569603 +0000 UTC m=+305.433793960" watchObservedRunningTime="2026-07-19 14:41:33.012376047 +0000 UTC m=+305.434600378" Jul 19 14:41:33.914718 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:33.914686 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:41:34.690805 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.690765 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-75spp"] Jul 19 14:41:34.694584 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.694563 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.697033 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.696999 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 19 14:41:34.697033 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.697024 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 19 14:41:34.697207 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.697122 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zt9dh\"" Jul 19 14:41:34.697334 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.697317 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 19 14:41:34.844267 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.844238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-tls\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.844267 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.844271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-textfile\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.844496 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.844291 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-wtmp\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.844496 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.844318 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebbcac45-6d88-4eaf-83f9-e1520f88b049-sys\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.844496 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.844380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebbcac45-6d88-4eaf-83f9-e1520f88b049-metrics-client-ca\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.844606 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.844523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-accelerators-collector-config\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.844606 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.844591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.844678 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.844647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqf2f\" (UniqueName: \"kubernetes.io/projected/ebbcac45-6d88-4eaf-83f9-e1520f88b049-kube-api-access-zqf2f\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.844712 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.844689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebbcac45-6d88-4eaf-83f9-e1520f88b049-root\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945534 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945453 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebbcac45-6d88-4eaf-83f9-e1520f88b049-metrics-client-ca\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945534 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945510 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-accelerators-collector-config\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945554 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945583 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqf2f\" (UniqueName: \"kubernetes.io/projected/ebbcac45-6d88-4eaf-83f9-e1520f88b049-kube-api-access-zqf2f\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebbcac45-6d88-4eaf-83f9-e1520f88b049-root\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945645 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-tls\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-textfile\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-wtmp\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945704 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebbcac45-6d88-4eaf-83f9-e1520f88b049-sys\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.945743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945728 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebbcac45-6d88-4eaf-83f9-e1520f88b049-root\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.946151 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.945755 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebbcac45-6d88-4eaf-83f9-e1520f88b049-sys\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.946200 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.946153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-wtmp\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.946200 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.946189 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-accelerators-collector-config\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.946371 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.946348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-textfile\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.946593 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.946573 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebbcac45-6d88-4eaf-83f9-e1520f88b049-metrics-client-ca\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.948039 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.948011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.948123 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.948012 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebbcac45-6d88-4eaf-83f9-e1520f88b049-node-exporter-tls\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:34.953408 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:34.953389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqf2f\" (UniqueName: \"kubernetes.io/projected/ebbcac45-6d88-4eaf-83f9-e1520f88b049-kube-api-access-zqf2f\") pod \"node-exporter-75spp\" (UID: \"ebbcac45-6d88-4eaf-83f9-e1520f88b049\") " pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:35.004407 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.004386 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75spp" Jul 19 14:41:35.016311 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:35.016286 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbcac45_6d88_4eaf_83f9_e1520f88b049.slice/crio-7ba957853d9758afa7eeb4cc01f62e52fe03e113377667bc1900c3ebbbf19e88 WatchSource:0}: Error finding container 7ba957853d9758afa7eeb4cc01f62e52fe03e113377667bc1900c3ebbbf19e88: Status 404 returned error can't find the container with id 7ba957853d9758afa7eeb4cc01f62e52fe03e113377667bc1900c3ebbbf19e88 Jul 19 14:41:35.746013 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.745965 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:41:35.750965 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.750941 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.754093 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.753677 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 19 14:41:35.754093 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.753724 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 19 14:41:35.754093 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.753929 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 19 14:41:35.754093 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.753959 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 19 14:41:35.754419 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.754231 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4wbr6\"" Jul 19 14:41:35.754419 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.754416 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 19 14:41:35.754594 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.754571 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 19 14:41:35.754740 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.754722 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 19 14:41:35.754821 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.754805 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 19 14:41:35.755115 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.754961 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 19 14:41:35.762062 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.761945 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:41:35.851563 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851530 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851563 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851757 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851585 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851757 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851757 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851757 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851757 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851756 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-web-config\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851962 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851782 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-config-volume\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851962 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851812 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851962 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851840 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-config-out\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851962 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851962 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x877h\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-kube-api-access-x877h\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.851962 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.851927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.952969 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.952936 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953147 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953147 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953120 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953147 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953136 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953307 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953155 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953307 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953307 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953209 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953307 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953249 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-web-config\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953307 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-config-volume\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953532 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953532 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953360 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-config-out\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953532 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953383 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953532 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953405 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x877h\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-kube-api-access-x877h\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.953991 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.953964 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.954595 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.954563 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.955369 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.955343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.958252 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.958125 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.959541 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.958625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-web-config\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.959636 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.959583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.959974 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.959929 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-config-volume\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.960300 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.960271 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.960603 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.960580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.960670 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.960605 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.960766 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.960741 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.961388 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.961364 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-config-out\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:35.962411 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:35.962389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x877h\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-kube-api-access-x877h\") pod \"alertmanager-main-0\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:36.008851 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:36.008706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75spp" event={"ID":"ebbcac45-6d88-4eaf-83f9-e1520f88b049","Type":"ContainerStarted","Data":"7ba957853d9758afa7eeb4cc01f62e52fe03e113377667bc1900c3ebbbf19e88"} Jul 19 14:41:36.067520 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:36.067475 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:41:36.351382 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:36.351352 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:41:36.352854 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:36.352830 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44765fea_4bf1_4363_82ef_0542d76eba0f.slice/crio-7e13709bdf67f7836f502db8f120d0eae31ac989974da83d01f76fc5742966fd WatchSource:0}: Error finding container 7e13709bdf67f7836f502db8f120d0eae31ac989974da83d01f76fc5742966fd: Status 404 returned error can't find the container with id 7e13709bdf67f7836f502db8f120d0eae31ac989974da83d01f76fc5742966fd Jul 19 14:41:37.013756 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:37.013713 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerStarted","Data":"7e13709bdf67f7836f502db8f120d0eae31ac989974da83d01f76fc5742966fd"} Jul 19 14:41:37.015711 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:37.015678 2559 generic.go:356] "Generic (PLEG): container finished" podID="ebbcac45-6d88-4eaf-83f9-e1520f88b049" containerID="1ce42d33b525063430938616dd94b4b53ce3ac505d05bafbf8598c0e9c2825b6" exitCode=0 Jul 19 14:41:37.015849 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:37.015722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75spp" event={"ID":"ebbcac45-6d88-4eaf-83f9-e1520f88b049","Type":"ContainerDied","Data":"1ce42d33b525063430938616dd94b4b53ce3ac505d05bafbf8598c0e9c2825b6"} Jul 19 14:41:37.101546 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:37.101524 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:41:37.739124 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:37.739095 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:37.739273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:37.739160 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:37.745265 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:37.745240 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:38.020996 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.020958 2559 generic.go:356] "Generic (PLEG): container finished" podID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerID="cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab" exitCode=0 Jul 19 14:41:38.021394 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.021044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerDied","Data":"cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab"} Jul 19 14:41:38.023275 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.023232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75spp" event={"ID":"ebbcac45-6d88-4eaf-83f9-e1520f88b049","Type":"ContainerStarted","Data":"a0d6bd8c8b80cd987c5e280283933d04d5d3b13dfd969eacb62702196c268f9c"} Jul 19 14:41:38.023373 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.023292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75spp" event={"ID":"ebbcac45-6d88-4eaf-83f9-e1520f88b049","Type":"ContainerStarted","Data":"6bc72cbab26e892514741f10f52caeebdc5189455f98ffc12adbee1f51972b4f"} Jul 19 14:41:38.026936 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.026920 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:41:38.967082 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.967024 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-75spp" podStartSLOduration=3.663323877 podStartE2EDuration="4.967003688s" totalImagesPullingTime="1.240786099s" totalInitContainerRuntime="62.893712ms" isStatefulPod=true podCreationTimestamp="2026-07-19 14:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:38.073218749 +0000 UTC m=+310.495443080" watchObservedRunningTime="2026-07-19 14:41:38.967003688 +0000 UTC m=+311.389228066" Jul 19 14:41:38.967259 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.967179 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76f4d76857-x8krn"] Jul 19 14:41:38.970323 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.970302 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:38.973669 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.973644 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5rvt9\"" Jul 19 14:41:38.973785 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.973674 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 19 14:41:38.973785 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.973716 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9hacnurk4lbip\"" Jul 19 14:41:38.973785 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.973674 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 19 14:41:38.973785 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.973777 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 19 14:41:38.974096 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.973835 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 19 14:41:38.979263 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:38.979227 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76f4d76857-x8krn"] Jul 19 14:41:39.079770 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.079731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-client-ca-bundle\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.080234 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.079781 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-secret-metrics-server-tls\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.080234 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.079884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-audit-log\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.080234 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.079927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.080234 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.079988 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pqc8\" (UniqueName: \"kubernetes.io/projected/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-kube-api-access-8pqc8\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.080234 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.080062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-metrics-server-audit-profiles\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.080234 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.080149 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-secret-metrics-server-client-certs\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.180743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.180702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-secret-metrics-server-client-certs\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.180943 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.180776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-client-ca-bundle\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.180943 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.180820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-secret-metrics-server-tls\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.180943 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.180921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-audit-log\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.181052 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.180949 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.181082 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.181049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pqc8\" (UniqueName: \"kubernetes.io/projected/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-kube-api-access-8pqc8\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.181230 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.181212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-metrics-server-audit-profiles\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.181699 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.181651 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.181988 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.181945 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-audit-log\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.182419 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.182361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-metrics-server-audit-profiles\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.184086 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.184062 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-secret-metrics-server-client-certs\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.184287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.184263 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-client-ca-bundle\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.184409 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.184386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-secret-metrics-server-tls\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.188657 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.188633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pqc8\" (UniqueName: \"kubernetes.io/projected/b6fb01f4-f68c-4d47-95d0-89728ace8e1c-kube-api-access-8pqc8\") pod \"metrics-server-76f4d76857-x8krn\" (UID: \"b6fb01f4-f68c-4d47-95d0-89728ace8e1c\") " pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.282269 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.282190 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:39.747841 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:39.747821 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76f4d76857-x8krn"] Jul 19 14:41:39.750815 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:41:39.750778 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fb01f4_f68c_4d47_95d0_89728ace8e1c.slice/crio-a5588f26c82bff6031b05872f169aa968f5a0797980ae92bb33a5384913964a6 WatchSource:0}: Error finding container a5588f26c82bff6031b05872f169aa968f5a0797980ae92bb33a5384913964a6: Status 404 returned error can't find the container with id a5588f26c82bff6031b05872f169aa968f5a0797980ae92bb33a5384913964a6 Jul 19 14:41:40.031833 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:40.031803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" event={"ID":"b6fb01f4-f68c-4d47-95d0-89728ace8e1c","Type":"ContainerStarted","Data":"a5588f26c82bff6031b05872f169aa968f5a0797980ae92bb33a5384913964a6"} Jul 19 14:41:40.034831 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:40.034803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerStarted","Data":"ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d"} Jul 19 14:41:40.034939 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:40.034903 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerStarted","Data":"150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57"} Jul 19 14:41:40.034980 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:40.034946 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerStarted","Data":"2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046"} Jul 19 14:41:40.035011 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:40.034986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerStarted","Data":"74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c"} Jul 19 14:41:40.035039 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:40.035023 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerStarted","Data":"40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b"} Jul 19 14:41:42.056411 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:42.056374 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerStarted","Data":"08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc"} Jul 19 14:41:42.057831 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:42.057807 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" event={"ID":"b6fb01f4-f68c-4d47-95d0-89728ace8e1c","Type":"ContainerStarted","Data":"566d617a29f25f11eb89a65990c113ec513ed01eed57a9b2986756a20d17e468"} Jul 19 14:41:42.087180 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:42.086092 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.834497754 podStartE2EDuration="7.086071853s" totalImagesPullingTime="4.236704829s" totalInitContainerRuntime="14.86927ms" isStatefulPod=false podCreationTimestamp="2026-07-19 14:41:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:42.083886626 +0000 UTC m=+314.506110958" watchObservedRunningTime="2026-07-19 14:41:42.086071853 +0000 UTC m=+314.508296186" Jul 19 14:41:42.108934 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:42.105727 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" podStartSLOduration=2.4500751960000002 podStartE2EDuration="4.105706195s" totalImagesPullingTime="1.655630999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:41:42.103655975 +0000 UTC m=+314.525880307" watchObservedRunningTime="2026-07-19 14:41:42.105706195 +0000 UTC m=+314.527930528" Jul 19 14:41:46.536377 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:46.536339 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666d9b4699-8pbzg"] Jul 19 14:41:59.282817 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:59.282762 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:41:59.283299 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:41:59.282900 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:42:02.090590 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:02.090557 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bb554f8b-qzhb7"] Jul 19 14:42:05.139756 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:05.139720 2559 generic.go:356] "Generic (PLEG): container finished" podID="4a0d9ee4-05f8-44f3-b282-f234b9bbeb13" containerID="3bfaecdb948d7ad1b7e1196dbd5fd3e287571677d161dbfe3225c711c10d8b5d" exitCode=0 Jul 19 14:42:05.140269 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:05.139779 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" event={"ID":"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13","Type":"ContainerDied","Data":"3bfaecdb948d7ad1b7e1196dbd5fd3e287571677d161dbfe3225c711c10d8b5d"} Jul 19 14:42:05.140269 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:05.140182 2559 scope.go:117] "RemoveContainer" containerID="3bfaecdb948d7ad1b7e1196dbd5fd3e287571677d161dbfe3225c711c10d8b5d" Jul 19 14:42:06.144710 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:06.144678 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6xlb" event={"ID":"4a0d9ee4-05f8-44f3-b282-f234b9bbeb13","Type":"ContainerStarted","Data":"1b11e63ea6a01aea3f9aecc45f4ded80835a9eaa6217c8f4b55389d53ad24c5c"} Jul 19 14:42:10.160252 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:10.160165 2559 generic.go:356] "Generic (PLEG): container finished" podID="65b96bc2-0209-4d7d-aa1c-a2348b7b14b9" containerID="20478defc216d05e9bb208525106dc346e79e81338eb983e3e54d401f27ffea3" exitCode=0 Jul 19 14:42:10.160252 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:10.160231 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sjhrl" event={"ID":"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9","Type":"ContainerDied","Data":"20478defc216d05e9bb208525106dc346e79e81338eb983e3e54d401f27ffea3"} Jul 19 14:42:10.160636 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:10.160539 2559 scope.go:117] "RemoveContainer" containerID="20478defc216d05e9bb208525106dc346e79e81338eb983e3e54d401f27ffea3" Jul 19 14:42:11.165982 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.165945 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sjhrl" event={"ID":"65b96bc2-0209-4d7d-aa1c-a2348b7b14b9","Type":"ContainerStarted","Data":"e18941b681a2564120622a4bb09f5f7bf7792addc3cc44da36043275046f6d89"} Jul 19 14:42:11.555855 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.555764 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" podUID="68cdeba4-359c-4409-8c6d-8297f763352b" containerName="registry" containerID="cri-o://c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a" gracePeriod=30 Jul 19 14:42:11.790993 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.790970 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:42:11.874995 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.874970 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") pod \"68cdeba4-359c-4409-8c6d-8297f763352b\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " Jul 19 14:42:11.875158 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.875009 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-installation-pull-secrets\") pod \"68cdeba4-359c-4409-8c6d-8297f763352b\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " Jul 19 14:42:11.875158 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.875042 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68cdeba4-359c-4409-8c6d-8297f763352b-ca-trust-extracted\") pod \"68cdeba4-359c-4409-8c6d-8297f763352b\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " Jul 19 14:42:11.875158 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.875082 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-registry-certificates\") pod \"68cdeba4-359c-4409-8c6d-8297f763352b\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " Jul 19 14:42:11.875158 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.875102 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-trusted-ca\") pod \"68cdeba4-359c-4409-8c6d-8297f763352b\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " Jul 19 14:42:11.875158 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.875121 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgfpd\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-kube-api-access-kgfpd\") pod \"68cdeba4-359c-4409-8c6d-8297f763352b\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " Jul 19 14:42:11.875158 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.875150 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-image-registry-private-configuration\") pod \"68cdeba4-359c-4409-8c6d-8297f763352b\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " Jul 19 14:42:11.875480 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.875175 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-bound-sa-token\") pod \"68cdeba4-359c-4409-8c6d-8297f763352b\" (UID: \"68cdeba4-359c-4409-8c6d-8297f763352b\") " Jul 19 14:42:11.875578 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.875547 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "68cdeba4-359c-4409-8c6d-8297f763352b" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:42:11.875650 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.875570 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "68cdeba4-359c-4409-8c6d-8297f763352b" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:42:11.877686 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.877643 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "68cdeba4-359c-4409-8c6d-8297f763352b" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:11.877686 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.877651 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-kube-api-access-kgfpd" (OuterVolumeSpecName: "kube-api-access-kgfpd") pod "68cdeba4-359c-4409-8c6d-8297f763352b" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b"). InnerVolumeSpecName "kube-api-access-kgfpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:42:11.877837 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.877644 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "68cdeba4-359c-4409-8c6d-8297f763352b" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:42:11.877929 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.877902 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "68cdeba4-359c-4409-8c6d-8297f763352b" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:11.877996 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.877980 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "68cdeba4-359c-4409-8c6d-8297f763352b" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:42:11.883453 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.883424 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cdeba4-359c-4409-8c6d-8297f763352b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "68cdeba4-359c-4409-8c6d-8297f763352b" (UID: "68cdeba4-359c-4409-8c6d-8297f763352b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:42:11.976818 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.976787 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-registry-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:11.976818 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.976814 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-installation-pull-secrets\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:11.977045 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.976829 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68cdeba4-359c-4409-8c6d-8297f763352b-ca-trust-extracted\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:11.977045 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.976843 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-registry-certificates\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:11.977045 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.976855 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68cdeba4-359c-4409-8c6d-8297f763352b-trusted-ca\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:11.977045 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.976881 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgfpd\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-kube-api-access-kgfpd\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:11.977045 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.976893 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68cdeba4-359c-4409-8c6d-8297f763352b-image-registry-private-configuration\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:11.977045 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:11.976905 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68cdeba4-359c-4409-8c6d-8297f763352b-bound-sa-token\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:12.170270 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:12.170178 2559 generic.go:356] "Generic (PLEG): container finished" podID="68cdeba4-359c-4409-8c6d-8297f763352b" containerID="c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a" exitCode=0 Jul 19 14:42:12.170270 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:12.170246 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" Jul 19 14:42:12.170270 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:12.170260 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" event={"ID":"68cdeba4-359c-4409-8c6d-8297f763352b","Type":"ContainerDied","Data":"c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a"} Jul 19 14:42:12.170748 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:12.170312 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d9b4699-8pbzg" event={"ID":"68cdeba4-359c-4409-8c6d-8297f763352b","Type":"ContainerDied","Data":"d4af97d41c6161baa3e6120abaf188b9b2ecab45d4e6c85753fe764d7e464d59"} Jul 19 14:42:12.170748 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:12.170342 2559 scope.go:117] "RemoveContainer" containerID="c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a" Jul 19 14:42:12.178381 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:12.178366 2559 scope.go:117] "RemoveContainer" containerID="c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a" Jul 19 14:42:12.178647 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:42:12.178611 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a\": container with ID starting with c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a not found: ID does not exist" containerID="c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a" Jul 19 14:42:12.178716 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:12.178653 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a"} err="failed to get container status \"c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a\": rpc error: code = NotFound desc = could not find container \"c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a\": container with ID starting with c18358ccfd2a0a7467ba7d7656619c82498c1008b29907f2dea6849a6a47862a not found: ID does not exist" Jul 19 14:42:12.185637 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:12.185618 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666d9b4699-8pbzg"] Jul 19 14:42:12.188933 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:12.188917 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-666d9b4699-8pbzg"] Jul 19 14:42:14.106555 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:14.106522 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68cdeba4-359c-4409-8c6d-8297f763352b" path="/var/lib/kubelet/pods/68cdeba4-359c-4409-8c6d-8297f763352b/volumes" Jul 19 14:42:19.287724 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:19.287692 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:42:19.291529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:19.291508 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76f4d76857-x8krn" Jul 19 14:42:27.112574 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.112476 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55bb554f8b-qzhb7" podUID="b09ee3a0-7b5d-4106-9d17-207120cc0aef" containerName="console" containerID="cri-o://6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d" gracePeriod=15 Jul 19 14:42:27.362931 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.362852 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:42:27.411205 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.411179 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-oauth-serving-cert\") pod \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " Jul 19 14:42:27.411348 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.411243 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-serving-cert\") pod \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " Jul 19 14:42:27.411348 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.411261 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-oauth-config\") pod \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " Jul 19 14:42:27.411348 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.411281 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4x72\" (UniqueName: \"kubernetes.io/projected/b09ee3a0-7b5d-4106-9d17-207120cc0aef-kube-api-access-h4x72\") pod \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " Jul 19 14:42:27.411348 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.411300 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-service-ca\") pod \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " Jul 19 14:42:27.411348 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.411321 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-config\") pod \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\" (UID: \"b09ee3a0-7b5d-4106-9d17-207120cc0aef\") " Jul 19 14:42:27.411732 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.411703 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b09ee3a0-7b5d-4106-9d17-207120cc0aef" (UID: "b09ee3a0-7b5d-4106-9d17-207120cc0aef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:42:27.411858 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.411754 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-service-ca" (OuterVolumeSpecName: "service-ca") pod "b09ee3a0-7b5d-4106-9d17-207120cc0aef" (UID: "b09ee3a0-7b5d-4106-9d17-207120cc0aef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:42:27.411858 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.411773 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-config" (OuterVolumeSpecName: "console-config") pod "b09ee3a0-7b5d-4106-9d17-207120cc0aef" (UID: "b09ee3a0-7b5d-4106-9d17-207120cc0aef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:42:27.413575 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.413548 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09ee3a0-7b5d-4106-9d17-207120cc0aef-kube-api-access-h4x72" (OuterVolumeSpecName: "kube-api-access-h4x72") pod "b09ee3a0-7b5d-4106-9d17-207120cc0aef" (UID: "b09ee3a0-7b5d-4106-9d17-207120cc0aef"). InnerVolumeSpecName "kube-api-access-h4x72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:42:27.413660 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.413570 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b09ee3a0-7b5d-4106-9d17-207120cc0aef" (UID: "b09ee3a0-7b5d-4106-9d17-207120cc0aef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:27.413660 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.413589 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b09ee3a0-7b5d-4106-9d17-207120cc0aef" (UID: "b09ee3a0-7b5d-4106-9d17-207120cc0aef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:27.512574 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.512537 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-oauth-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:27.512574 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.512571 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:27.512574 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.512580 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-oauth-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:27.512574 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.512589 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4x72\" (UniqueName: \"kubernetes.io/projected/b09ee3a0-7b5d-4106-9d17-207120cc0aef-kube-api-access-h4x72\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:27.512775 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.512598 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-service-ca\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:27.512775 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:27.512607 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b09ee3a0-7b5d-4106-9d17-207120cc0aef-console-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:28.231089 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:28.231053 2559 generic.go:356] "Generic (PLEG): container finished" podID="b09ee3a0-7b5d-4106-9d17-207120cc0aef" containerID="6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d" exitCode=0 Jul 19 14:42:28.231529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:28.231106 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bb554f8b-qzhb7" Jul 19 14:42:28.231529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:28.231132 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bb554f8b-qzhb7" event={"ID":"b09ee3a0-7b5d-4106-9d17-207120cc0aef","Type":"ContainerDied","Data":"6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d"} Jul 19 14:42:28.231529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:28.231186 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bb554f8b-qzhb7" event={"ID":"b09ee3a0-7b5d-4106-9d17-207120cc0aef","Type":"ContainerDied","Data":"a5e801424d67990bc51602ed211f1e6e1307aec7a6ef13785b080a9fd3725c47"} Jul 19 14:42:28.231529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:28.231208 2559 scope.go:117] "RemoveContainer" containerID="6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d" Jul 19 14:42:28.238995 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:28.238977 2559 scope.go:117] "RemoveContainer" containerID="6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d" Jul 19 14:42:28.239232 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:42:28.239214 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d\": container with ID starting with 6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d not found: ID does not exist" containerID="6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d" Jul 19 14:42:28.239290 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:28.239240 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d"} err="failed to get container status \"6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d\": rpc error: code = NotFound desc = could not find container \"6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d\": container with ID starting with 6b6c05aedecf53d3d5eba408a4ca47173df0581a95cb828ede52df86df72511d not found: ID does not exist" Jul 19 14:42:28.247196 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:28.247174 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bb554f8b-qzhb7"] Jul 19 14:42:28.250902 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:28.250882 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55bb554f8b-qzhb7"] Jul 19 14:42:30.105362 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:30.105331 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b09ee3a0-7b5d-4106-9d17-207120cc0aef" path="/var/lib/kubelet/pods/b09ee3a0-7b5d-4106-9d17-207120cc0aef/volumes" Jul 19 14:42:42.040560 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:42.040518 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:42:42.042897 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:42.042857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82fc003a-678c-4e3d-90ab-0c64f693f817-metrics-certs\") pod \"network-metrics-daemon-zw8ds\" (UID: \"82fc003a-678c-4e3d-90ab-0c64f693f817\") " pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:42:42.205329 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:42.205302 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5r6f\"" Jul 19 14:42:42.213238 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:42.213223 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zw8ds" Jul 19 14:42:42.335079 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:42.335056 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zw8ds"] Jul 19 14:42:42.337142 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:42:42.337118 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fc003a_678c_4e3d_90ab_0c64f693f817.slice/crio-fc31e336f3b426161566b743cfb77eef9d806c30ee90b139112264a9cece153f WatchSource:0}: Error finding container fc31e336f3b426161566b743cfb77eef9d806c30ee90b139112264a9cece153f: Status 404 returned error can't find the container with id fc31e336f3b426161566b743cfb77eef9d806c30ee90b139112264a9cece153f Jul 19 14:42:43.286817 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:43.286771 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zw8ds" event={"ID":"82fc003a-678c-4e3d-90ab-0c64f693f817","Type":"ContainerStarted","Data":"fc31e336f3b426161566b743cfb77eef9d806c30ee90b139112264a9cece153f"} Jul 19 14:42:44.291624 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:44.291590 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zw8ds" event={"ID":"82fc003a-678c-4e3d-90ab-0c64f693f817","Type":"ContainerStarted","Data":"393383fb46be48a1ec327818f3e26253ebecafb5fe4032fcb4bfd7a5d217954f"} Jul 19 14:42:44.292117 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:44.291654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zw8ds" event={"ID":"82fc003a-678c-4e3d-90ab-0c64f693f817","Type":"ContainerStarted","Data":"8b380e4882007c1143537f54d33e1c8cf90ddb944ca95d4cf24578d9bc112a5a"} Jul 19 14:42:44.306371 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:44.306321 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zw8ds" podStartSLOduration=375.088050927 podStartE2EDuration="6m16.306304222s" totalImagesPullingTime="1.218253295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:42:44.305754985 +0000 UTC m=+376.727979327" watchObservedRunningTime="2026-07-19 14:42:44.306304222 +0000 UTC m=+376.728528555" Jul 19 14:42:55.003948 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.003916 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:42:55.004362 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.004188 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="alertmanager" containerID="cri-o://40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b" gracePeriod=120 Jul 19 14:42:55.004362 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.004255 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy-metric" containerID="cri-o://ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d" gracePeriod=120 Jul 19 14:42:55.004362 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.004273 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy-web" containerID="cri-o://2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046" gracePeriod=120 Jul 19 14:42:55.004362 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.004308 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="config-reloader" containerID="cri-o://74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c" gracePeriod=120 Jul 19 14:42:55.004576 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.004327 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="prom-label-proxy" containerID="cri-o://08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc" gracePeriod=120 Jul 19 14:42:55.004576 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.004457 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy" containerID="cri-o://150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57" gracePeriod=120 Jul 19 14:42:55.335515 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.335480 2559 generic.go:356] "Generic (PLEG): container finished" podID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerID="08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc" exitCode=0 Jul 19 14:42:55.335515 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.335507 2559 generic.go:356] "Generic (PLEG): container finished" podID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerID="150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57" exitCode=0 Jul 19 14:42:55.335515 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.335515 2559 generic.go:356] "Generic (PLEG): container finished" podID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerID="74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c" exitCode=0 Jul 19 14:42:55.335515 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.335521 2559 generic.go:356] "Generic (PLEG): container finished" podID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerID="40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b" exitCode=0 Jul 19 14:42:55.335826 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.335557 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerDied","Data":"08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc"} Jul 19 14:42:55.335826 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.335641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerDied","Data":"150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57"} Jul 19 14:42:55.335826 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.335682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerDied","Data":"74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c"} Jul 19 14:42:55.335826 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:55.335718 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerDied","Data":"40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b"} Jul 19 14:42:56.248718 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.248696 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.342622 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.342592 2559 generic.go:356] "Generic (PLEG): container finished" podID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerID="ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d" exitCode=0 Jul 19 14:42:56.342622 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.342618 2559 generic.go:356] "Generic (PLEG): container finished" podID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerID="2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046" exitCode=0 Jul 19 14:42:56.342807 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.342672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerDied","Data":"ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d"} Jul 19 14:42:56.342807 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.342741 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerDied","Data":"2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046"} Jul 19 14:42:56.342807 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.342684 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.342807 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.342778 2559 scope.go:117] "RemoveContainer" containerID="08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc" Jul 19 14:42:56.342983 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.342769 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44765fea-4bf1-4363-82ef-0542d76eba0f","Type":"ContainerDied","Data":"7e13709bdf67f7836f502db8f120d0eae31ac989974da83d01f76fc5742966fd"} Jul 19 14:42:56.349908 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.349892 2559 scope.go:117] "RemoveContainer" containerID="ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d" Jul 19 14:42:56.356213 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.356196 2559 scope.go:117] "RemoveContainer" containerID="150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57" Jul 19 14:42:56.359306 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.359277 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-cluster-tls-config\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.359390 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.359318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-config-out\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.359390 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.359334 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-main-tls\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.359390 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.359360 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.359540 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.359393 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-trusted-ca-bundle\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.359540 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.359417 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-tls-assets\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.359795 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.359770 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:42:56.359907 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.359848 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x877h\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-kube-api-access-x877h\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.360007 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.359987 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-metrics-client-ca\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.360215 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.360197 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-web\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.361715 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.361544 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-main-db\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.361715 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.361586 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-web-config\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.361715 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.360505 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:42:56.361715 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.361623 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-config-volume\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.361957 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.361789 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy\") pod \"44765fea-4bf1-4363-82ef-0542d76eba0f\" (UID: \"44765fea-4bf1-4363-82ef-0542d76eba0f\") " Jul 19 14:42:56.361957 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.361806 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:42:56.362560 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.362083 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-main-db\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.362560 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.362122 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.362560 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.362137 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44765fea-4bf1-4363-82ef-0542d76eba0f-metrics-client-ca\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.362767 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.362609 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-config-out" (OuterVolumeSpecName: "config-out") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:42:56.363242 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.363208 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:56.363421 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.363389 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:56.363499 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.363471 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:56.363558 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.363490 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-kube-api-access-x877h" (OuterVolumeSpecName: "kube-api-access-x877h") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "kube-api-access-x877h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:42:56.364007 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.363982 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:42:56.364188 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.364165 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-config-volume" (OuterVolumeSpecName: "config-volume") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:56.365170 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.365147 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:56.365578 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.365562 2559 scope.go:117] "RemoveContainer" containerID="2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046" Jul 19 14:42:56.366723 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.366610 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:56.376081 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.376055 2559 scope.go:117] "RemoveContainer" containerID="74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c" Jul 19 14:42:56.376176 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.376048 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-web-config" (OuterVolumeSpecName: "web-config") pod "44765fea-4bf1-4363-82ef-0542d76eba0f" (UID: "44765fea-4bf1-4363-82ef-0542d76eba0f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:42:56.382710 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.382694 2559 scope.go:117] "RemoveContainer" containerID="40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b" Jul 19 14:42:56.389441 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.389425 2559 scope.go:117] "RemoveContainer" containerID="cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab" Jul 19 14:42:56.395984 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.395958 2559 scope.go:117] "RemoveContainer" containerID="08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc" Jul 19 14:42:56.396211 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:42:56.396192 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc\": container with ID starting with 08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc not found: ID does not exist" containerID="08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc" Jul 19 14:42:56.396285 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.396216 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc"} err="failed to get container status \"08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc\": rpc error: code = NotFound desc = could not find container \"08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc\": container with ID starting with 08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc not found: ID does not exist" Jul 19 14:42:56.396285 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.396234 2559 scope.go:117] "RemoveContainer" containerID="ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d" Jul 19 14:42:56.396462 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:42:56.396445 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d\": container with ID starting with ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d not found: ID does not exist" containerID="ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d" Jul 19 14:42:56.396513 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.396467 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d"} err="failed to get container status \"ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d\": rpc error: code = NotFound desc = could not find container \"ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d\": container with ID starting with ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d not found: ID does not exist" Jul 19 14:42:56.396513 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.396483 2559 scope.go:117] "RemoveContainer" containerID="150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57" Jul 19 14:42:56.396704 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:42:56.396688 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57\": container with ID starting with 150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57 not found: ID does not exist" containerID="150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57" Jul 19 14:42:56.396768 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.396706 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57"} err="failed to get container status \"150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57\": rpc error: code = NotFound desc = could not find container \"150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57\": container with ID starting with 150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57 not found: ID does not exist" Jul 19 14:42:56.396768 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.396719 2559 scope.go:117] "RemoveContainer" containerID="2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046" Jul 19 14:42:56.396906 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:42:56.396882 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046\": container with ID starting with 2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046 not found: ID does not exist" containerID="2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046" Jul 19 14:42:56.396961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.396902 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046"} err="failed to get container status \"2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046\": rpc error: code = NotFound desc = could not find container \"2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046\": container with ID starting with 2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046 not found: ID does not exist" Jul 19 14:42:56.396961 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.396918 2559 scope.go:117] "RemoveContainer" containerID="74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c" Jul 19 14:42:56.397138 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:42:56.397119 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c\": container with ID starting with 74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c not found: ID does not exist" containerID="74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c" Jul 19 14:42:56.397209 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.397140 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c"} err="failed to get container status \"74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c\": rpc error: code = NotFound desc = could not find container \"74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c\": container with ID starting with 74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c not found: ID does not exist" Jul 19 14:42:56.397209 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.397154 2559 scope.go:117] "RemoveContainer" containerID="40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b" Jul 19 14:42:56.397396 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:42:56.397377 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b\": container with ID starting with 40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b not found: ID does not exist" containerID="40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b" Jul 19 14:42:56.397458 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.397399 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b"} err="failed to get container status \"40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b\": rpc error: code = NotFound desc = could not find container \"40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b\": container with ID starting with 40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b not found: ID does not exist" Jul 19 14:42:56.397458 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.397414 2559 scope.go:117] "RemoveContainer" containerID="cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab" Jul 19 14:42:56.397642 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:42:56.397626 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab\": container with ID starting with cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab not found: ID does not exist" containerID="cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab" Jul 19 14:42:56.397705 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.397645 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab"} err="failed to get container status \"cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab\": rpc error: code = NotFound desc = could not find container \"cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab\": container with ID starting with cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab not found: ID does not exist" Jul 19 14:42:56.397705 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.397657 2559 scope.go:117] "RemoveContainer" containerID="08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc" Jul 19 14:42:56.397846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.397829 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc"} err="failed to get container status \"08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc\": rpc error: code = NotFound desc = could not find container \"08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc\": container with ID starting with 08b74939df2aac6a8c36dc79aaa7ce0dd258b33429d2a6642b9b83bfbba879bc not found: ID does not exist" Jul 19 14:42:56.397846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.397846 2559 scope.go:117] "RemoveContainer" containerID="ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d" Jul 19 14:42:56.398080 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.398059 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d"} err="failed to get container status \"ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d\": rpc error: code = NotFound desc = could not find container \"ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d\": container with ID starting with ae2159c3616ad1ca83259091e931b09ec414aa61be737804cd6fd61486dd610d not found: ID does not exist" Jul 19 14:42:56.398080 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.398078 2559 scope.go:117] "RemoveContainer" containerID="150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57" Jul 19 14:42:56.398328 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.398305 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57"} err="failed to get container status \"150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57\": rpc error: code = NotFound desc = could not find container \"150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57\": container with ID starting with 150c0674de3000e511906ef41d871bb1fccabeb81a1c3953ea4e6968e0775e57 not found: ID does not exist" Jul 19 14:42:56.398404 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.398329 2559 scope.go:117] "RemoveContainer" containerID="2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046" Jul 19 14:42:56.398556 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.398540 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046"} err="failed to get container status \"2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046\": rpc error: code = NotFound desc = could not find container \"2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046\": container with ID starting with 2894e3da5a62ec3e4159e6e0aa93ba1df5c0f37073d454a7d2ab24049b10b046 not found: ID does not exist" Jul 19 14:42:56.398597 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.398556 2559 scope.go:117] "RemoveContainer" containerID="74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c" Jul 19 14:42:56.398775 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.398759 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c"} err="failed to get container status \"74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c\": rpc error: code = NotFound desc = could not find container \"74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c\": container with ID starting with 74f74ad15416d807df2d62440ab2d896b4d377756640973758e2aae8f09f0f1c not found: ID does not exist" Jul 19 14:42:56.398820 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.398775 2559 scope.go:117] "RemoveContainer" containerID="40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b" Jul 19 14:42:56.399011 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.398995 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b"} err="failed to get container status \"40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b\": rpc error: code = NotFound desc = could not find container \"40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b\": container with ID starting with 40bd05f4cbe243c1c5ad69357d763725d654e31fef522ec391d3af5fe036843b not found: ID does not exist" Jul 19 14:42:56.399011 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.399011 2559 scope.go:117] "RemoveContainer" containerID="cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab" Jul 19 14:42:56.399232 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.399210 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab"} err="failed to get container status \"cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab\": rpc error: code = NotFound desc = could not find container \"cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab\": container with ID starting with cc4578e89d990e0dd9ccdec9476e872c5d7047b0bc573b2e1cc801a686bb29ab not found: ID does not exist" Jul 19 14:42:56.463003 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.462980 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.463090 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.463003 2559 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-cluster-tls-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.463090 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.463017 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44765fea-4bf1-4363-82ef-0542d76eba0f-config-out\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.463090 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.463026 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-main-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.463090 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.463047 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.463090 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.463056 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-tls-assets\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.463090 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.463066 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x877h\" (UniqueName: \"kubernetes.io/projected/44765fea-4bf1-4363-82ef-0542d76eba0f-kube-api-access-x877h\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.463090 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.463075 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.463090 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.463087 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-web-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.463336 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.463100 2559 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44765fea-4bf1-4363-82ef-0542d76eba0f-config-volume\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:42:56.665626 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.665588 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:42:56.670157 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.670131 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:42:56.700401 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700375 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:42:56.700683 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700666 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68cdeba4-359c-4409-8c6d-8297f763352b" containerName="registry" Jul 19 14:42:56.700755 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700686 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68cdeba4-359c-4409-8c6d-8297f763352b" containerName="registry" Jul 19 14:42:56.700755 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700696 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy-metric" Jul 19 14:42:56.700755 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700705 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy-metric" Jul 19 14:42:56.700755 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700718 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="config-reloader" Jul 19 14:42:56.700755 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700728 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="config-reloader" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700738 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy-web" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700771 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy-web" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700794 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="init-config-reloader" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700804 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="init-config-reloader" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700816 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09ee3a0-7b5d-4106-9d17-207120cc0aef" containerName="console" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700824 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09ee3a0-7b5d-4106-9d17-207120cc0aef" containerName="console" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700833 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="alertmanager" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700841 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="alertmanager" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700851 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700859 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700884 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="prom-label-proxy" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700892 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="prom-label-proxy" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700969 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="config-reloader" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700981 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.700992 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="prom-label-proxy" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.701003 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy-web" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.701012 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b09ee3a0-7b5d-4106-9d17-207120cc0aef" containerName="console" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.701020 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="alertmanager" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.701028 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68cdeba4-359c-4409-8c6d-8297f763352b" containerName="registry" Jul 19 14:42:56.701026 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.701035 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" containerName="kube-rbac-proxy-metric" Jul 19 14:42:56.706393 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.706373 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.709125 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.708912 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 19 14:42:56.709125 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.708924 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 19 14:42:56.709125 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.708940 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 19 14:42:56.709125 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.708999 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 19 14:42:56.709125 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.708946 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4wbr6\"" Jul 19 14:42:56.709125 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.709017 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 19 14:42:56.709125 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.709047 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 19 14:42:56.709125 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.709000 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 19 14:42:56.709445 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.709188 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 19 14:42:56.714816 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.714794 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 19 14:42:56.720142 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.720124 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:42:56.866994 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.866958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.866994 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.866996 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867247 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867030 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6ccf501-4751-47b9-9573-1e6cdf51748c-config-out\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867247 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867247 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867176 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6ccf501-4751-47b9-9573-1e6cdf51748c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867247 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6ccf501-4751-47b9-9573-1e6cdf51748c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867247 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867210 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867247 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czd9k\" (UniqueName: \"kubernetes.io/projected/b6ccf501-4751-47b9-9573-1e6cdf51748c-kube-api-access-czd9k\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867247 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867247 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-web-config\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867337 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6ccf501-4751-47b9-9573-1e6cdf51748c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867378 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.867530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.867405 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6ccf501-4751-47b9-9573-1e6cdf51748c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968131 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968131 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6ccf501-4751-47b9-9573-1e6cdf51748c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968131 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6ccf501-4751-47b9-9573-1e6cdf51748c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968131 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968132 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968446 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czd9k\" (UniqueName: \"kubernetes.io/projected/b6ccf501-4751-47b9-9573-1e6cdf51748c-kube-api-access-czd9k\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968446 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968446 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968311 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-web-config\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968446 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968338 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6ccf501-4751-47b9-9573-1e6cdf51748c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968446 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968365 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968446 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968389 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6ccf501-4751-47b9-9573-1e6cdf51748c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.968886 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.968831 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6ccf501-4751-47b9-9573-1e6cdf51748c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.969043 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.969022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6ccf501-4751-47b9-9573-1e6cdf51748c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.969116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.969076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.969116 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.969109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.969221 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.969147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6ccf501-4751-47b9-9573-1e6cdf51748c-config-out\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.969275 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.969227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6ccf501-4751-47b9-9573-1e6cdf51748c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.971413 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.971391 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-web-config\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.971530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.971444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.971678 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.971657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6ccf501-4751-47b9-9573-1e6cdf51748c-config-out\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.971992 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.971974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.972063 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.972041 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.972063 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.972051 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6ccf501-4751-47b9-9573-1e6cdf51748c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.972136 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.972048 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.972136 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.972086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.972738 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.972723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6ccf501-4751-47b9-9573-1e6cdf51748c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:56.976599 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:56.976575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czd9k\" (UniqueName: \"kubernetes.io/projected/b6ccf501-4751-47b9-9573-1e6cdf51748c-kube-api-access-czd9k\") pod \"alertmanager-main-0\" (UID: \"b6ccf501-4751-47b9-9573-1e6cdf51748c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:57.016307 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:57.016275 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:42:57.151393 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:57.151360 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:42:57.154158 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:42:57.154132 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ccf501_4751_47b9_9573_1e6cdf51748c.slice/crio-ba1b6913d8211e1fffeaf5b9f8a73d772ff6fc09a7281281fa54f8acd547266a WatchSource:0}: Error finding container ba1b6913d8211e1fffeaf5b9f8a73d772ff6fc09a7281281fa54f8acd547266a: Status 404 returned error can't find the container with id ba1b6913d8211e1fffeaf5b9f8a73d772ff6fc09a7281281fa54f8acd547266a Jul 19 14:42:57.348426 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:57.348388 2559 generic.go:356] "Generic (PLEG): container finished" podID="b6ccf501-4751-47b9-9573-1e6cdf51748c" containerID="a457ea0834f0caaa1312f1b88f25b17ae1f32041bd904de3a174ba481725f9c1" exitCode=0 Jul 19 14:42:57.348768 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:57.348462 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6ccf501-4751-47b9-9573-1e6cdf51748c","Type":"ContainerDied","Data":"a457ea0834f0caaa1312f1b88f25b17ae1f32041bd904de3a174ba481725f9c1"} Jul 19 14:42:57.348768 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:57.348523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6ccf501-4751-47b9-9573-1e6cdf51748c","Type":"ContainerStarted","Data":"ba1b6913d8211e1fffeaf5b9f8a73d772ff6fc09a7281281fa54f8acd547266a"} Jul 19 14:42:58.106915 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:58.106859 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44765fea-4bf1-4363-82ef-0542d76eba0f" path="/var/lib/kubelet/pods/44765fea-4bf1-4363-82ef-0542d76eba0f/volumes" Jul 19 14:42:58.355773 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:58.355693 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6ccf501-4751-47b9-9573-1e6cdf51748c","Type":"ContainerStarted","Data":"f44dd56b24d2c7027559cedf95d7084cb830b8226a1421b1a9848f19193dd251"} Jul 19 14:42:58.356163 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:58.355778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6ccf501-4751-47b9-9573-1e6cdf51748c","Type":"ContainerStarted","Data":"61cb54eaaa9527440f3d18a432234fb6723244449eaf105fc73d08705c27620a"} Jul 19 14:42:58.356163 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:58.355806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6ccf501-4751-47b9-9573-1e6cdf51748c","Type":"ContainerStarted","Data":"4c70616480a05c718f6f439bd956f30682f23b2e040e3b3e7129bf2293e6f825"} Jul 19 14:42:58.356163 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:58.355829 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6ccf501-4751-47b9-9573-1e6cdf51748c","Type":"ContainerStarted","Data":"d7eb7e29f6db620d2a917c2a05d8007d9eda63f139ecba39d4cecdf55d10f557"} Jul 19 14:42:58.356163 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:58.355853 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6ccf501-4751-47b9-9573-1e6cdf51748c","Type":"ContainerStarted","Data":"50d9be4d8861889b4f81ed357072792a188036d9d959143ecb20066c6ebc4ee3"} Jul 19 14:42:58.356163 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:58.355896 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6ccf501-4751-47b9-9573-1e6cdf51748c","Type":"ContainerStarted","Data":"ace54751a995dc7ec3eac9389b031794a617286d4897592e670c770a7c010db5"} Jul 19 14:42:58.382638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:42:58.382587 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.362455654 podStartE2EDuration="2.382567438s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.111784ms" isStatefulPod=false podCreationTimestamp="2026-07-19 14:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:42:58.381350814 +0000 UTC m=+390.803575147" watchObservedRunningTime="2026-07-19 14:42:58.382567438 +0000 UTC m=+390.804791771" Jul 19 14:43:11.878221 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:43:11.878169 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" podUID="30c83ee5-7890-4240-957c-49de880b15a4" Jul 19 14:43:12.408039 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:43:12.408010 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:43:14.111655 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:43:14.111619 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:43:14.114055 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:43:14.114022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30c83ee5-7890-4240-957c-49de880b15a4-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2dcp2\" (UID: \"30c83ee5-7890-4240-957c-49de880b15a4\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:43:14.212237 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:43:14.212197 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nfdzm\"" Jul 19 14:43:14.220100 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:43:14.220085 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" Jul 19 14:43:14.345758 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:43:14.345731 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2"] Jul 19 14:43:14.347930 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:43:14.347903 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c83ee5_7890_4240_957c_49de880b15a4.slice/crio-eaa97a1581ef702ac159774c6d5cf95df5eb4ceb5d27ad6d44f307d505b6d428 WatchSource:0}: Error finding container eaa97a1581ef702ac159774c6d5cf95df5eb4ceb5d27ad6d44f307d505b6d428: Status 404 returned error can't find the container with id eaa97a1581ef702ac159774c6d5cf95df5eb4ceb5d27ad6d44f307d505b6d428 Jul 19 14:43:14.416259 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:43:14.416175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" event={"ID":"30c83ee5-7890-4240-957c-49de880b15a4","Type":"ContainerStarted","Data":"eaa97a1581ef702ac159774c6d5cf95df5eb4ceb5d27ad6d44f307d505b6d428"} Jul 19 14:43:16.426895 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:43:16.426841 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" event={"ID":"30c83ee5-7890-4240-957c-49de880b15a4","Type":"ContainerStarted","Data":"af4b5c163fa33e3dce3435e1ed87371b2d33a64abcca0fa58c6d73e561c055f8"} Jul 19 14:43:16.442098 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:43:16.442025 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2dcp2" podStartSLOduration=383.36504021 podStartE2EDuration="6m24.44200818s" totalImagesPullingTime="1.07696797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:43:16.44192271 +0000 UTC m=+408.864147043" watchObservedRunningTime="2026-07-19 14:43:16.44200818 +0000 UTC m=+408.864232514" Jul 19 14:44:15.731956 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.731916 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l"] Jul 19 14:44:15.735532 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.735511 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:15.738195 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.738173 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 19 14:44:15.739093 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.739074 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zh9qx\"" Jul 19 14:44:15.739198 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.739073 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 19 14:44:15.745638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.745615 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l"] Jul 19 14:44:15.879740 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.879695 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:15.879964 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.879758 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:15.879964 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.879816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbr8\" (UniqueName: \"kubernetes.io/projected/e291e518-0e54-4f91-8507-a16ab3a8193c-kube-api-access-7kbr8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:15.980526 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.980491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:15.980691 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.980550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:15.980691 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.980578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbr8\" (UniqueName: \"kubernetes.io/projected/e291e518-0e54-4f91-8507-a16ab3a8193c-kube-api-access-7kbr8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:15.980928 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.980907 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:15.980977 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.980958 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:15.988525 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:15.988459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbr8\" (UniqueName: \"kubernetes.io/projected/e291e518-0e54-4f91-8507-a16ab3a8193c-kube-api-access-7kbr8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:16.045152 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:16.045103 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:16.174286 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:16.174216 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l"] Jul 19 14:44:16.176447 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:44:16.176423 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode291e518_0e54_4f91_8507_a16ab3a8193c.slice/crio-8402755b261dd4400cd0bcbae8084b1577f35e853d18716ded94b18a1b55e02d WatchSource:0}: Error finding container 8402755b261dd4400cd0bcbae8084b1577f35e853d18716ded94b18a1b55e02d: Status 404 returned error can't find the container with id 8402755b261dd4400cd0bcbae8084b1577f35e853d18716ded94b18a1b55e02d Jul 19 14:44:16.647431 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:16.647399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" event={"ID":"e291e518-0e54-4f91-8507-a16ab3a8193c","Type":"ContainerStarted","Data":"8402755b261dd4400cd0bcbae8084b1577f35e853d18716ded94b18a1b55e02d"} Jul 19 14:44:23.677641 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:23.677604 2559 generic.go:356] "Generic (PLEG): container finished" podID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerID="3a21d524cf49ee793926b1b7574a6389ffabbe8eafa5577e04b3fb50fba8f9d5" exitCode=0 Jul 19 14:44:23.678025 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:23.677684 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" event={"ID":"e291e518-0e54-4f91-8507-a16ab3a8193c","Type":"ContainerDied","Data":"3a21d524cf49ee793926b1b7574a6389ffabbe8eafa5577e04b3fb50fba8f9d5"} Jul 19 14:44:26.696048 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:26.696012 2559 generic.go:356] "Generic (PLEG): container finished" podID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerID="37d6062543d56672869b82413b7c93f9069a851ac8cdc010f905997195b91b1f" exitCode=0 Jul 19 14:44:26.696434 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:26.696096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" event={"ID":"e291e518-0e54-4f91-8507-a16ab3a8193c","Type":"ContainerDied","Data":"37d6062543d56672869b82413b7c93f9069a851ac8cdc010f905997195b91b1f"} Jul 19 14:44:34.735310 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:34.735257 2559 generic.go:356] "Generic (PLEG): container finished" podID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerID="9043db856b54e60aca45a42084bf736c51f2ac7ee0a7587ae8eec33d5988e525" exitCode=0 Jul 19 14:44:34.735686 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:34.735344 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" event={"ID":"e291e518-0e54-4f91-8507-a16ab3a8193c","Type":"ContainerDied","Data":"9043db856b54e60aca45a42084bf736c51f2ac7ee0a7587ae8eec33d5988e525"} Jul 19 14:44:35.859243 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:35.859218 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:35.934288 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:35.934258 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-bundle\") pod \"e291e518-0e54-4f91-8507-a16ab3a8193c\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " Jul 19 14:44:35.934417 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:35.934303 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kbr8\" (UniqueName: \"kubernetes.io/projected/e291e518-0e54-4f91-8507-a16ab3a8193c-kube-api-access-7kbr8\") pod \"e291e518-0e54-4f91-8507-a16ab3a8193c\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " Jul 19 14:44:35.934417 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:35.934329 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-util\") pod \"e291e518-0e54-4f91-8507-a16ab3a8193c\" (UID: \"e291e518-0e54-4f91-8507-a16ab3a8193c\") " Jul 19 14:44:35.934979 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:35.934948 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-bundle" (OuterVolumeSpecName: "bundle") pod "e291e518-0e54-4f91-8507-a16ab3a8193c" (UID: "e291e518-0e54-4f91-8507-a16ab3a8193c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:44:35.936452 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:35.936424 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e291e518-0e54-4f91-8507-a16ab3a8193c-kube-api-access-7kbr8" (OuterVolumeSpecName: "kube-api-access-7kbr8") pod "e291e518-0e54-4f91-8507-a16ab3a8193c" (UID: "e291e518-0e54-4f91-8507-a16ab3a8193c"). InnerVolumeSpecName "kube-api-access-7kbr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:44:35.938580 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:35.938559 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-util" (OuterVolumeSpecName: "util") pod "e291e518-0e54-4f91-8507-a16ab3a8193c" (UID: "e291e518-0e54-4f91-8507-a16ab3a8193c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:44:36.035637 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:36.035574 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:44:36.035637 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:36.035598 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kbr8\" (UniqueName: \"kubernetes.io/projected/e291e518-0e54-4f91-8507-a16ab3a8193c-kube-api-access-7kbr8\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:44:36.035637 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:36.035609 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e291e518-0e54-4f91-8507-a16ab3a8193c-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:44:36.744146 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:36.744113 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" Jul 19 14:44:36.744146 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:36.744122 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405gq6l" event={"ID":"e291e518-0e54-4f91-8507-a16ab3a8193c","Type":"ContainerDied","Data":"8402755b261dd4400cd0bcbae8084b1577f35e853d18716ded94b18a1b55e02d"} Jul 19 14:44:36.744353 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:36.744171 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8402755b261dd4400cd0bcbae8084b1577f35e853d18716ded94b18a1b55e02d" Jul 19 14:44:43.720011 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.719981 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k"] Jul 19 14:44:43.720439 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.720246 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerName="extract" Jul 19 14:44:43.720439 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.720257 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerName="extract" Jul 19 14:44:43.720439 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.720277 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerName="util" Jul 19 14:44:43.720439 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.720282 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerName="util" Jul 19 14:44:43.720439 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.720292 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerName="pull" Jul 19 14:44:43.720439 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.720298 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerName="pull" Jul 19 14:44:43.720439 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.720337 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e291e518-0e54-4f91-8507-a16ab3a8193c" containerName="extract" Jul 19 14:44:43.723707 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.723690 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" Jul 19 14:44:43.726698 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.726674 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 19 14:44:43.726829 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.726816 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:44:43.726907 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.726854 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hvpr7\"" Jul 19 14:44:43.734664 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.734642 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k"] Jul 19 14:44:43.787721 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.787693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm44t\" (UniqueName: \"kubernetes.io/projected/5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf-kube-api-access-gm44t\") pod \"cert-manager-operator-controller-manager-55c444bf86-tb24k\" (UID: \"5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" Jul 19 14:44:43.787852 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.787739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tb24k\" (UID: \"5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" Jul 19 14:44:43.888648 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.888618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm44t\" (UniqueName: \"kubernetes.io/projected/5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf-kube-api-access-gm44t\") pod \"cert-manager-operator-controller-manager-55c444bf86-tb24k\" (UID: \"5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" Jul 19 14:44:43.888781 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.888663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tb24k\" (UID: \"5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" Jul 19 14:44:43.889018 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.889003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tb24k\" (UID: \"5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" Jul 19 14:44:43.897451 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:43.897423 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm44t\" (UniqueName: \"kubernetes.io/projected/5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf-kube-api-access-gm44t\") pod \"cert-manager-operator-controller-manager-55c444bf86-tb24k\" (UID: \"5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" Jul 19 14:44:44.032946 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:44.032860 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" Jul 19 14:44:44.164916 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:44.164891 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k"] Jul 19 14:44:44.167082 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:44:44.167060 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d02dc6e_c6a0_4c13_a016_85e4a3b8a6cf.slice/crio-f53203b56b44a813f68773fab91acb2e48954ba97b2eaf6bacf03f48555d9a5d WatchSource:0}: Error finding container f53203b56b44a813f68773fab91acb2e48954ba97b2eaf6bacf03f48555d9a5d: Status 404 returned error can't find the container with id f53203b56b44a813f68773fab91acb2e48954ba97b2eaf6bacf03f48555d9a5d Jul 19 14:44:44.780839 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:44.780801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" event={"ID":"5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf","Type":"ContainerStarted","Data":"f53203b56b44a813f68773fab91acb2e48954ba97b2eaf6bacf03f48555d9a5d"} Jul 19 14:44:46.792182 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:46.792143 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" event={"ID":"5d02dc6e-c6a0-4c13-a016-85e4a3b8a6cf","Type":"ContainerStarted","Data":"5eabfb3759c6f9840e17bd70bd406efb61734cab15628b6e6beb719678fd47c6"} Jul 19 14:44:46.813315 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:46.813262 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tb24k" podStartSLOduration=2.16133126 podStartE2EDuration="3.813245462s" totalImagesPullingTime="1.651914202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:44:46.811601539 +0000 UTC m=+499.233825871" watchObservedRunningTime="2026-07-19 14:44:46.813245462 +0000 UTC m=+499.235469793" Jul 19 14:44:47.496550 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.496520 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25"] Jul 19 14:44:47.499165 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.499141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.501813 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.501794 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zh9qx\"" Jul 19 14:44:47.501937 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.501919 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 19 14:44:47.502838 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.502821 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 19 14:44:47.507949 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.507926 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25"] Jul 19 14:44:47.520140 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.520115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzdvt\" (UniqueName: \"kubernetes.io/projected/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-kube-api-access-tzdvt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.520225 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.520182 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.520225 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.520204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.621172 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.621148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.621287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.621182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.621287 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.621207 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdvt\" (UniqueName: \"kubernetes.io/projected/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-kube-api-access-tzdvt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.621495 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.621476 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.621559 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.621516 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.641674 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.641654 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzdvt\" (UniqueName: \"kubernetes.io/projected/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-kube-api-access-tzdvt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.809152 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.809096 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:47.931441 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.931418 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25"] Jul 19 14:44:47.934046 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:44:47.934015 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4be361_17a5_4a9d_a84c_fee8e1ce0bce.slice/crio-46480dcdaf01402e1a22e8d4705eba4a1ed7beb81af94c1eb5fdd4a9d711e83c WatchSource:0}: Error finding container 46480dcdaf01402e1a22e8d4705eba4a1ed7beb81af94c1eb5fdd4a9d711e83c: Status 404 returned error can't find the container with id 46480dcdaf01402e1a22e8d4705eba4a1ed7beb81af94c1eb5fdd4a9d711e83c Jul 19 14:44:47.976201 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.976166 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ghbzt"] Jul 19 14:44:47.979505 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.979488 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:44:47.982153 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.982134 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xjtq9\"" Jul 19 14:44:47.982253 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.982168 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 19 14:44:47.982354 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.982341 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 19 14:44:47.989031 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:47.989009 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ghbzt"] Jul 19 14:44:48.025894 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.025854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2lxt\" (UniqueName: \"kubernetes.io/projected/bed0e8de-ea27-4201-98d0-160fd32adfce-kube-api-access-s2lxt\") pod \"cert-manager-webhook-6486ffd997-ghbzt\" (UID: \"bed0e8de-ea27-4201-98d0-160fd32adfce\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:44:48.026074 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.025911 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bed0e8de-ea27-4201-98d0-160fd32adfce-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ghbzt\" (UID: \"bed0e8de-ea27-4201-98d0-160fd32adfce\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:44:48.126376 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.126348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lxt\" (UniqueName: \"kubernetes.io/projected/bed0e8de-ea27-4201-98d0-160fd32adfce-kube-api-access-s2lxt\") pod \"cert-manager-webhook-6486ffd997-ghbzt\" (UID: \"bed0e8de-ea27-4201-98d0-160fd32adfce\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:44:48.126501 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.126392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bed0e8de-ea27-4201-98d0-160fd32adfce-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ghbzt\" (UID: \"bed0e8de-ea27-4201-98d0-160fd32adfce\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:44:48.136879 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.136849 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bed0e8de-ea27-4201-98d0-160fd32adfce-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ghbzt\" (UID: \"bed0e8de-ea27-4201-98d0-160fd32adfce\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:44:48.136968 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.136941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2lxt\" (UniqueName: \"kubernetes.io/projected/bed0e8de-ea27-4201-98d0-160fd32adfce-kube-api-access-s2lxt\") pod \"cert-manager-webhook-6486ffd997-ghbzt\" (UID: \"bed0e8de-ea27-4201-98d0-160fd32adfce\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:44:48.299005 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.298978 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:44:48.421426 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.421370 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ghbzt"] Jul 19 14:44:48.423451 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:44:48.423421 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbed0e8de_ea27_4201_98d0_160fd32adfce.slice/crio-1fe10ad47055b5c73a177227e063f23b12641396b6a2a24fe6904bb7180031f1 WatchSource:0}: Error finding container 1fe10ad47055b5c73a177227e063f23b12641396b6a2a24fe6904bb7180031f1: Status 404 returned error can't find the container with id 1fe10ad47055b5c73a177227e063f23b12641396b6a2a24fe6904bb7180031f1 Jul 19 14:44:48.801200 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.801103 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" event={"ID":"bed0e8de-ea27-4201-98d0-160fd32adfce","Type":"ContainerStarted","Data":"1fe10ad47055b5c73a177227e063f23b12641396b6a2a24fe6904bb7180031f1"} Jul 19 14:44:48.802514 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.802488 2559 generic.go:356] "Generic (PLEG): container finished" podID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerID="86f06eefa02aa0487411eef783874310cd7aace029fa2f054150d7076aec37ec" exitCode=0 Jul 19 14:44:48.802648 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.802536 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" event={"ID":"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce","Type":"ContainerDied","Data":"86f06eefa02aa0487411eef783874310cd7aace029fa2f054150d7076aec37ec"} Jul 19 14:44:48.802648 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:48.802567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" event={"ID":"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce","Type":"ContainerStarted","Data":"46480dcdaf01402e1a22e8d4705eba4a1ed7beb81af94c1eb5fdd4a9d711e83c"} Jul 19 14:44:51.819489 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:51.819456 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" event={"ID":"bed0e8de-ea27-4201-98d0-160fd32adfce","Type":"ContainerStarted","Data":"abd0ce5c0cb775988d3cb011c58e95e0267238816beb281dc990479ba87b66a5"} Jul 19 14:44:51.821515 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:51.821491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" event={"ID":"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce","Type":"ContainerStarted","Data":"7a1fc572e650ba17d59cfb5467a73a9e19ba75654daddbdd9f4bf139e3f3085b"} Jul 19 14:44:52.827724 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:52.827687 2559 generic.go:356] "Generic (PLEG): container finished" podID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerID="7a1fc572e650ba17d59cfb5467a73a9e19ba75654daddbdd9f4bf139e3f3085b" exitCode=0 Jul 19 14:44:52.828137 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:52.827773 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" event={"ID":"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce","Type":"ContainerDied","Data":"7a1fc572e650ba17d59cfb5467a73a9e19ba75654daddbdd9f4bf139e3f3085b"} Jul 19 14:44:52.828137 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:52.827944 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:44:52.858240 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:52.858183 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" podStartSLOduration=2.533233836 podStartE2EDuration="5.858167787s" totalImagesPullingTime="3.324933951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:44:52.857228578 +0000 UTC m=+505.279452907" watchObservedRunningTime="2026-07-19 14:44:52.858167787 +0000 UTC m=+505.280392118" Jul 19 14:44:53.834360 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:53.834326 2559 generic.go:356] "Generic (PLEG): container finished" podID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerID="7f11bffdc147aa9560858517125d3a28b6450217f5fcdc7626dc6b8dfecdecb6" exitCode=0 Jul 19 14:44:53.834838 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:53.834409 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" event={"ID":"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce","Type":"ContainerDied","Data":"7f11bffdc147aa9560858517125d3a28b6450217f5fcdc7626dc6b8dfecdecb6"} Jul 19 14:44:54.959036 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:54.959007 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:55.083207 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.083170 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-bundle\") pod \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " Jul 19 14:44:55.083369 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.083228 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzdvt\" (UniqueName: \"kubernetes.io/projected/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-kube-api-access-tzdvt\") pod \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " Jul 19 14:44:55.083369 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.083267 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-util\") pod \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\" (UID: \"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce\") " Jul 19 14:44:55.083617 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.083593 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-bundle" (OuterVolumeSpecName: "bundle") pod "2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" (UID: "2e4be361-17a5-4a9d-a84c-fee8e1ce0bce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:44:55.085203 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.085177 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-kube-api-access-tzdvt" (OuterVolumeSpecName: "kube-api-access-tzdvt") pod "2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" (UID: "2e4be361-17a5-4a9d-a84c-fee8e1ce0bce"). InnerVolumeSpecName "kube-api-access-tzdvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:44:55.087602 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.087569 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-util" (OuterVolumeSpecName: "util") pod "2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" (UID: "2e4be361-17a5-4a9d-a84c-fee8e1ce0bce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:44:55.183940 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.183917 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:44:55.183940 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.183939 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:44:55.184065 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.183948 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzdvt\" (UniqueName: \"kubernetes.io/projected/2e4be361-17a5-4a9d-a84c-fee8e1ce0bce-kube-api-access-tzdvt\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:44:55.844264 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.844227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" event={"ID":"2e4be361-17a5-4a9d-a84c-fee8e1ce0bce","Type":"ContainerDied","Data":"46480dcdaf01402e1a22e8d4705eba4a1ed7beb81af94c1eb5fdd4a9d711e83c"} Jul 19 14:44:55.844264 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.844252 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fn7f25" Jul 19 14:44:55.844469 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:55.844280 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46480dcdaf01402e1a22e8d4705eba4a1ed7beb81af94c1eb5fdd4a9d711e83c" Jul 19 14:44:58.836975 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:44:58.836943 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ghbzt" Jul 19 14:45:01.204197 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.204158 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k"] Jul 19 14:45:01.204568 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.204466 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerName="extract" Jul 19 14:45:01.204568 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.204479 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerName="extract" Jul 19 14:45:01.204568 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.204490 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerName="pull" Jul 19 14:45:01.204568 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.204496 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerName="pull" Jul 19 14:45:01.204568 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.204506 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerName="util" Jul 19 14:45:01.204568 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.204512 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerName="util" Jul 19 14:45:01.204568 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.204559 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4be361-17a5-4a9d-a84c-fee8e1ce0bce" containerName="extract" Jul 19 14:45:01.208056 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.208038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" Jul 19 14:45:01.210470 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.210447 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 19 14:45:01.210571 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.210456 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ttc57\"" Jul 19 14:45:01.210571 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.210491 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:45:01.216593 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.216570 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k"] Jul 19 14:45:01.332159 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.332128 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c73e9315-c657-478c-a100-7f7c1a5bd5d7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dh65k\" (UID: \"c73e9315-c657-478c-a100-7f7c1a5bd5d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" Jul 19 14:45:01.332159 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.332163 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz7c8\" (UniqueName: \"kubernetes.io/projected/c73e9315-c657-478c-a100-7f7c1a5bd5d7-kube-api-access-nz7c8\") pod \"openshift-lws-operator-fd8ccff4c-dh65k\" (UID: \"c73e9315-c657-478c-a100-7f7c1a5bd5d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" Jul 19 14:45:01.432696 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.432661 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c73e9315-c657-478c-a100-7f7c1a5bd5d7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dh65k\" (UID: \"c73e9315-c657-478c-a100-7f7c1a5bd5d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" Jul 19 14:45:01.432696 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.432696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz7c8\" (UniqueName: \"kubernetes.io/projected/c73e9315-c657-478c-a100-7f7c1a5bd5d7-kube-api-access-nz7c8\") pod \"openshift-lws-operator-fd8ccff4c-dh65k\" (UID: \"c73e9315-c657-478c-a100-7f7c1a5bd5d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" Jul 19 14:45:01.433106 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.433084 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c73e9315-c657-478c-a100-7f7c1a5bd5d7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dh65k\" (UID: \"c73e9315-c657-478c-a100-7f7c1a5bd5d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" Jul 19 14:45:01.441508 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.441483 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz7c8\" (UniqueName: \"kubernetes.io/projected/c73e9315-c657-478c-a100-7f7c1a5bd5d7-kube-api-access-nz7c8\") pod \"openshift-lws-operator-fd8ccff4c-dh65k\" (UID: \"c73e9315-c657-478c-a100-7f7c1a5bd5d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" Jul 19 14:45:01.516786 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.516719 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" Jul 19 14:45:01.639586 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.639563 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k"] Jul 19 14:45:01.642221 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:45:01.642192 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc73e9315_c657_478c_a100_7f7c1a5bd5d7.slice/crio-e2d0bbc783c8c5595abf50b2df4f0dad40d4deb66b1d37ce7a20999b2253f954 WatchSource:0}: Error finding container e2d0bbc783c8c5595abf50b2df4f0dad40d4deb66b1d37ce7a20999b2253f954: Status 404 returned error can't find the container with id e2d0bbc783c8c5595abf50b2df4f0dad40d4deb66b1d37ce7a20999b2253f954 Jul 19 14:45:01.869915 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:01.869884 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" event={"ID":"c73e9315-c657-478c-a100-7f7c1a5bd5d7","Type":"ContainerStarted","Data":"e2d0bbc783c8c5595abf50b2df4f0dad40d4deb66b1d37ce7a20999b2253f954"} Jul 19 14:45:03.884718 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:03.884686 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" event={"ID":"c73e9315-c657-478c-a100-7f7c1a5bd5d7","Type":"ContainerStarted","Data":"4a579b60ed98b0bfd835f5719dc70c3197eef312ea103c95e98446fc774bae6b"} Jul 19 14:45:03.900534 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:03.900495 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dh65k" podStartSLOduration=0.873528064 podStartE2EDuration="2.900481737s" totalImagesPullingTime="2.026953673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:45:03.899159991 +0000 UTC m=+516.321384325" watchObservedRunningTime="2026-07-19 14:45:03.900481737 +0000 UTC m=+516.322706067" Jul 19 14:45:06.643969 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.643933 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm"] Jul 19 14:45:06.647690 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.647672 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.650209 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.650186 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 19 14:45:06.651073 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.651053 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zh9qx\"" Jul 19 14:45:06.651175 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.651053 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 19 14:45:06.655309 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.655290 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm"] Jul 19 14:45:06.775844 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.775804 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkmbz\" (UniqueName: \"kubernetes.io/projected/245659a6-561f-457a-b5a4-5b5a539cece0-kube-api-access-kkmbz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.776029 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.775856 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.776029 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.775948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.876425 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.876391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkmbz\" (UniqueName: \"kubernetes.io/projected/245659a6-561f-457a-b5a4-5b5a539cece0-kube-api-access-kkmbz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.876563 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.876432 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.876563 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.876470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.876807 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.876791 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.876841 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.876821 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.884684 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.884662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkmbz\" (UniqueName: \"kubernetes.io/projected/245659a6-561f-457a-b5a4-5b5a539cece0-kube-api-access-kkmbz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:06.957102 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:06.957039 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:07.093714 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:07.093689 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm"] Jul 19 14:45:07.096065 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:45:07.096035 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245659a6_561f_457a_b5a4_5b5a539cece0.slice/crio-1d649c5c9aaae4b76294db9a9a807dd5ecdab5dd34fe5e15dac942ae9d47e3e3 WatchSource:0}: Error finding container 1d649c5c9aaae4b76294db9a9a807dd5ecdab5dd34fe5e15dac942ae9d47e3e3: Status 404 returned error can't find the container with id 1d649c5c9aaae4b76294db9a9a807dd5ecdab5dd34fe5e15dac942ae9d47e3e3 Jul 19 14:45:07.905009 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:07.904978 2559 generic.go:356] "Generic (PLEG): container finished" podID="245659a6-561f-457a-b5a4-5b5a539cece0" containerID="5e262a42ca1eb1949ee8dbba631f5168ce14785f182e14fbbaa0b73b911d062d" exitCode=0 Jul 19 14:45:07.905402 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:07.905073 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" event={"ID":"245659a6-561f-457a-b5a4-5b5a539cece0","Type":"ContainerDied","Data":"5e262a42ca1eb1949ee8dbba631f5168ce14785f182e14fbbaa0b73b911d062d"} Jul 19 14:45:07.905402 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:07.905127 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" event={"ID":"245659a6-561f-457a-b5a4-5b5a539cece0","Type":"ContainerStarted","Data":"1d649c5c9aaae4b76294db9a9a807dd5ecdab5dd34fe5e15dac942ae9d47e3e3"} Jul 19 14:45:09.915819 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:09.915785 2559 generic.go:356] "Generic (PLEG): container finished" podID="245659a6-561f-457a-b5a4-5b5a539cece0" containerID="80cbfe3a1aaf702019605f84efe16c2eb54d7086b30e92f8f2c1c285fcbf71ed" exitCode=0 Jul 19 14:45:09.916232 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:09.915833 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" event={"ID":"245659a6-561f-457a-b5a4-5b5a539cece0","Type":"ContainerDied","Data":"80cbfe3a1aaf702019605f84efe16c2eb54d7086b30e92f8f2c1c285fcbf71ed"} Jul 19 14:45:10.922209 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:10.922173 2559 generic.go:356] "Generic (PLEG): container finished" podID="245659a6-561f-457a-b5a4-5b5a539cece0" containerID="96ab6da423b57d7e2995612ef2310cdd95eb63271851188a402dcf702358b250" exitCode=0 Jul 19 14:45:10.922623 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:10.922206 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" event={"ID":"245659a6-561f-457a-b5a4-5b5a539cece0","Type":"ContainerDied","Data":"96ab6da423b57d7e2995612ef2310cdd95eb63271851188a402dcf702358b250"} Jul 19 14:45:12.056298 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.056273 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:12.121801 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.121771 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-util\") pod \"245659a6-561f-457a-b5a4-5b5a539cece0\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " Jul 19 14:45:12.121969 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.121839 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkmbz\" (UniqueName: \"kubernetes.io/projected/245659a6-561f-457a-b5a4-5b5a539cece0-kube-api-access-kkmbz\") pod \"245659a6-561f-457a-b5a4-5b5a539cece0\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " Jul 19 14:45:12.121969 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.121878 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-bundle\") pod \"245659a6-561f-457a-b5a4-5b5a539cece0\" (UID: \"245659a6-561f-457a-b5a4-5b5a539cece0\") " Jul 19 14:45:12.122616 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.122591 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-bundle" (OuterVolumeSpecName: "bundle") pod "245659a6-561f-457a-b5a4-5b5a539cece0" (UID: "245659a6-561f-457a-b5a4-5b5a539cece0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:45:12.123956 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.123927 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245659a6-561f-457a-b5a4-5b5a539cece0-kube-api-access-kkmbz" (OuterVolumeSpecName: "kube-api-access-kkmbz") pod "245659a6-561f-457a-b5a4-5b5a539cece0" (UID: "245659a6-561f-457a-b5a4-5b5a539cece0"). InnerVolumeSpecName "kube-api-access-kkmbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:45:12.127751 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.127725 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-util" (OuterVolumeSpecName: "util") pod "245659a6-561f-457a-b5a4-5b5a539cece0" (UID: "245659a6-561f-457a-b5a4-5b5a539cece0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:45:12.223254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.223178 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:45:12.223254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.223201 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkmbz\" (UniqueName: \"kubernetes.io/projected/245659a6-561f-457a-b5a4-5b5a539cece0-kube-api-access-kkmbz\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:45:12.223254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.223211 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245659a6-561f-457a-b5a4-5b5a539cece0-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:45:12.932318 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.932286 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" Jul 19 14:45:12.932484 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.932285 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpvxnm" event={"ID":"245659a6-561f-457a-b5a4-5b5a539cece0","Type":"ContainerDied","Data":"1d649c5c9aaae4b76294db9a9a807dd5ecdab5dd34fe5e15dac942ae9d47e3e3"} Jul 19 14:45:12.932484 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:12.932406 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d649c5c9aaae4b76294db9a9a807dd5ecdab5dd34fe5e15dac942ae9d47e3e3" Jul 19 14:45:22.956203 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.956115 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7499d794d8-497rt"] Jul 19 14:45:22.956638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.956412 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245659a6-561f-457a-b5a4-5b5a539cece0" containerName="extract" Jul 19 14:45:22.956638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.956423 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="245659a6-561f-457a-b5a4-5b5a539cece0" containerName="extract" Jul 19 14:45:22.956638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.956432 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245659a6-561f-457a-b5a4-5b5a539cece0" containerName="util" Jul 19 14:45:22.956638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.956437 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="245659a6-561f-457a-b5a4-5b5a539cece0" containerName="util" Jul 19 14:45:22.956638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.956447 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245659a6-561f-457a-b5a4-5b5a539cece0" containerName="pull" Jul 19 14:45:22.956638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.956453 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="245659a6-561f-457a-b5a4-5b5a539cece0" containerName="pull" Jul 19 14:45:22.956638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.956501 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="245659a6-561f-457a-b5a4-5b5a539cece0" containerName="extract" Jul 19 14:45:22.959231 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.959213 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:22.963095 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.963065 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 19 14:45:22.964117 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.964102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nkcpj\"" Jul 19 14:45:22.964224 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.964175 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 19 14:45:22.964279 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.964227 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 19 14:45:22.985024 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:22.985001 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7499d794d8-497rt"] Jul 19 14:45:23.110319 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.110277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/58079111-f079-4364-8c69-47bff67e612f-manager-config\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.110516 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.110350 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58079111-f079-4364-8c69-47bff67e612f-cert\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.110516 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.110390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbth9\" (UniqueName: \"kubernetes.io/projected/58079111-f079-4364-8c69-47bff67e612f-kube-api-access-zbth9\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.110516 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.110424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/58079111-f079-4364-8c69-47bff67e612f-metrics-cert\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.211743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.211656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58079111-f079-4364-8c69-47bff67e612f-cert\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.211743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.211700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbth9\" (UniqueName: \"kubernetes.io/projected/58079111-f079-4364-8c69-47bff67e612f-kube-api-access-zbth9\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.211743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.211732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/58079111-f079-4364-8c69-47bff67e612f-metrics-cert\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.212059 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.211787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/58079111-f079-4364-8c69-47bff67e612f-manager-config\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.212478 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.212446 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/58079111-f079-4364-8c69-47bff67e612f-manager-config\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.214247 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.214223 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/58079111-f079-4364-8c69-47bff67e612f-metrics-cert\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.214361 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.214244 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58079111-f079-4364-8c69-47bff67e612f-cert\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.229103 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.229084 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbth9\" (UniqueName: \"kubernetes.io/projected/58079111-f079-4364-8c69-47bff67e612f-kube-api-access-zbth9\") pod \"lws-controller-manager-7499d794d8-497rt\" (UID: \"58079111-f079-4364-8c69-47bff67e612f\") " pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.268183 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.268159 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:23.438862 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:45:23.438826 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58079111_f079_4364_8c69_47bff67e612f.slice/crio-be976ff19ebb5bc7d8d66d669bf452f32c2e96c92d118f1c955d7c6adfcc491b WatchSource:0}: Error finding container be976ff19ebb5bc7d8d66d669bf452f32c2e96c92d118f1c955d7c6adfcc491b: Status 404 returned error can't find the container with id be976ff19ebb5bc7d8d66d669bf452f32c2e96c92d118f1c955d7c6adfcc491b Jul 19 14:45:23.440338 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.440311 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7499d794d8-497rt"] Jul 19 14:45:23.979640 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:23.979596 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" event={"ID":"58079111-f079-4364-8c69-47bff67e612f","Type":"ContainerStarted","Data":"be976ff19ebb5bc7d8d66d669bf452f32c2e96c92d118f1c955d7c6adfcc491b"} Jul 19 14:45:24.600988 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.600949 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56"] Jul 19 14:45:24.604447 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.604424 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.606921 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.606891 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 19 14:45:24.607119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.606966 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 19 14:45:24.607119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.606985 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 19 14:45:24.607119 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.607056 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 19 14:45:24.607268 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.607177 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vts5g\"" Jul 19 14:45:24.617316 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.617295 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56"] Jul 19 14:45:24.724654 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.724610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb24w\" (UniqueName: \"kubernetes.io/projected/30b720f1-59a3-4905-9a2c-5bd386c81d12-kube-api-access-bb24w\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.724822 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.724673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/30b720f1-59a3-4905-9a2c-5bd386c81d12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.724822 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.724754 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/30b720f1-59a3-4905-9a2c-5bd386c81d12-manifests\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.724941 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.724840 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b720f1-59a3-4905-9a2c-5bd386c81d12-tmp\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.724941 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.724881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30b720f1-59a3-4905-9a2c-5bd386c81d12-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.825548 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.825507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb24w\" (UniqueName: \"kubernetes.io/projected/30b720f1-59a3-4905-9a2c-5bd386c81d12-kube-api-access-bb24w\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.825548 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.825555 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/30b720f1-59a3-4905-9a2c-5bd386c81d12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.825772 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.825584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/30b720f1-59a3-4905-9a2c-5bd386c81d12-manifests\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.825772 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.825632 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b720f1-59a3-4905-9a2c-5bd386c81d12-tmp\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.825772 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.825651 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30b720f1-59a3-4905-9a2c-5bd386c81d12-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.826175 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.826146 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/30b720f1-59a3-4905-9a2c-5bd386c81d12-manifests\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.828471 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.828449 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b720f1-59a3-4905-9a2c-5bd386c81d12-tmp\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.828558 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.828507 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/30b720f1-59a3-4905-9a2c-5bd386c81d12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.828623 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.828600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30b720f1-59a3-4905-9a2c-5bd386c81d12-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.833154 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.833133 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb24w\" (UniqueName: \"kubernetes.io/projected/30b720f1-59a3-4905-9a2c-5bd386c81d12-kube-api-access-bb24w\") pod \"opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56\" (UID: \"30b720f1-59a3-4905-9a2c-5bd386c81d12\") " pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:24.917652 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:24.917567 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:25.099420 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:25.099351 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56"] Jul 19 14:45:25.102392 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:45:25.102360 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b720f1_59a3_4905_9a2c_5bd386c81d12.slice/crio-04702936194b796869130f12baa8f3e71da5e2ff527173cb2102f0c49b8771b7 WatchSource:0}: Error finding container 04702936194b796869130f12baa8f3e71da5e2ff527173cb2102f0c49b8771b7: Status 404 returned error can't find the container with id 04702936194b796869130f12baa8f3e71da5e2ff527173cb2102f0c49b8771b7 Jul 19 14:45:25.993053 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:25.993014 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" event={"ID":"30b720f1-59a3-4905-9a2c-5bd386c81d12","Type":"ContainerStarted","Data":"04702936194b796869130f12baa8f3e71da5e2ff527173cb2102f0c49b8771b7"} Jul 19 14:45:25.995308 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:25.994994 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" event={"ID":"58079111-f079-4364-8c69-47bff67e612f","Type":"ContainerStarted","Data":"ddd3d2dca8566b4b67d7a8fa2c19ee19fb0506328f2dec55bf5078be2bc3d9e9"} Jul 19 14:45:25.995308 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:25.995240 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:26.011649 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:26.011573 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" podStartSLOduration=2.440760037 podStartE2EDuration="4.011555842s" totalImagesPullingTime="1.570795805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:45:26.011311414 +0000 UTC m=+538.433535745" watchObservedRunningTime="2026-07-19 14:45:26.011555842 +0000 UTC m=+538.433780174" Jul 19 14:45:29.012899 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:29.012839 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" event={"ID":"30b720f1-59a3-4905-9a2c-5bd386c81d12","Type":"ContainerStarted","Data":"315f4190f2e1bcc8befcac846dc51b81d87ce2e524c994e63b9ea75ef7b13742"} Jul 19 14:45:30.019067 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:30.019026 2559 generic.go:356] "Generic (PLEG): container finished" podID="30b720f1-59a3-4905-9a2c-5bd386c81d12" containerID="315f4190f2e1bcc8befcac846dc51b81d87ce2e524c994e63b9ea75ef7b13742" exitCode=0 Jul 19 14:45:30.019463 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:30.019086 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" event={"ID":"30b720f1-59a3-4905-9a2c-5bd386c81d12","Type":"ContainerDied","Data":"315f4190f2e1bcc8befcac846dc51b81d87ce2e524c994e63b9ea75ef7b13742"} Jul 19 14:45:31.025072 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:31.025030 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" event={"ID":"30b720f1-59a3-4905-9a2c-5bd386c81d12","Type":"ContainerStarted","Data":"7a0df7cc1a70326934b2f573e238b9c4a9a1dc7bf04430abb478b1b018bfc8d1"} Jul 19 14:45:31.025444 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:31.025184 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:31.046745 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:31.046684 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" podStartSLOduration=2.684857735 podStartE2EDuration="7.046662848s" totalImagesPullingTime="3.281956459s" totalInitContainerRuntime="1.079848654s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:45:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:45:31.045322347 +0000 UTC m=+543.467546702" watchObservedRunningTime="2026-07-19 14:45:31.046662848 +0000 UTC m=+543.468887191" Jul 19 14:45:37.003640 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:37.003608 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7499d794d8-497rt" Jul 19 14:45:42.037439 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:42.037404 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56" Jul 19 14:45:45.038931 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.038897 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv"] Jul 19 14:45:45.047757 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.047733 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.050203 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.050175 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 19 14:45:45.050333 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.050230 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dkv8t\"" Jul 19 14:45:45.050333 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.050273 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 19 14:45:45.058216 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.058194 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv"] Jul 19 14:45:45.198374 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.198332 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e021d53-1b40-42a6-8889-2fc1c5674287-tls-certs\") pod \"kube-auth-proxy-6fc68cb667-vt9qv\" (UID: \"2e021d53-1b40-42a6-8889-2fc1c5674287\") " pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.198522 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.198395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94nm8\" (UniqueName: \"kubernetes.io/projected/2e021d53-1b40-42a6-8889-2fc1c5674287-kube-api-access-94nm8\") pod \"kube-auth-proxy-6fc68cb667-vt9qv\" (UID: \"2e021d53-1b40-42a6-8889-2fc1c5674287\") " pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.198522 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.198464 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e021d53-1b40-42a6-8889-2fc1c5674287-tmp\") pod \"kube-auth-proxy-6fc68cb667-vt9qv\" (UID: \"2e021d53-1b40-42a6-8889-2fc1c5674287\") " pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.299979 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.299853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e021d53-1b40-42a6-8889-2fc1c5674287-tmp\") pod \"kube-auth-proxy-6fc68cb667-vt9qv\" (UID: \"2e021d53-1b40-42a6-8889-2fc1c5674287\") " pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.300148 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.300046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e021d53-1b40-42a6-8889-2fc1c5674287-tls-certs\") pod \"kube-auth-proxy-6fc68cb667-vt9qv\" (UID: \"2e021d53-1b40-42a6-8889-2fc1c5674287\") " pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.300223 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.300133 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94nm8\" (UniqueName: \"kubernetes.io/projected/2e021d53-1b40-42a6-8889-2fc1c5674287-kube-api-access-94nm8\") pod \"kube-auth-proxy-6fc68cb667-vt9qv\" (UID: \"2e021d53-1b40-42a6-8889-2fc1c5674287\") " pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.303216 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.303189 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e021d53-1b40-42a6-8889-2fc1c5674287-tmp\") pod \"kube-auth-proxy-6fc68cb667-vt9qv\" (UID: \"2e021d53-1b40-42a6-8889-2fc1c5674287\") " pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.303693 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.303676 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e021d53-1b40-42a6-8889-2fc1c5674287-tls-certs\") pod \"kube-auth-proxy-6fc68cb667-vt9qv\" (UID: \"2e021d53-1b40-42a6-8889-2fc1c5674287\") " pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.309403 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.309376 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94nm8\" (UniqueName: \"kubernetes.io/projected/2e021d53-1b40-42a6-8889-2fc1c5674287-kube-api-access-94nm8\") pod \"kube-auth-proxy-6fc68cb667-vt9qv\" (UID: \"2e021d53-1b40-42a6-8889-2fc1c5674287\") " pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.358973 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.358937 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" Jul 19 14:45:45.502515 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:45.502473 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv"] Jul 19 14:45:45.504746 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:45:45.504723 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e021d53_1b40_42a6_8889_2fc1c5674287.slice/crio-c747c61814b035e2361c4f6cc08e0d5f4c4e7d9a53857f731af0bb370f68c52c WatchSource:0}: Error finding container c747c61814b035e2361c4f6cc08e0d5f4c4e7d9a53857f731af0bb370f68c52c: Status 404 returned error can't find the container with id c747c61814b035e2361c4f6cc08e0d5f4c4e7d9a53857f731af0bb370f68c52c Jul 19 14:45:46.108534 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:46.108491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" event={"ID":"2e021d53-1b40-42a6-8889-2fc1c5674287","Type":"ContainerStarted","Data":"c747c61814b035e2361c4f6cc08e0d5f4c4e7d9a53857f731af0bb370f68c52c"} Jul 19 14:45:49.122989 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:49.122951 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" event={"ID":"2e021d53-1b40-42a6-8889-2fc1c5674287","Type":"ContainerStarted","Data":"53dd834b3809577b3bfcc45316e6264fa128c85f57210580cb9a9ce7c719de8f"} Jul 19 14:45:49.139445 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:45:49.139393 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fc68cb667-vt9qv" podStartSLOduration=1.584537394 podStartE2EDuration="4.139377651s" totalImagesPullingTime="2.554840257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:45:49.137814711 +0000 UTC m=+561.560039043" watchObservedRunningTime="2026-07-19 14:45:49.139377651 +0000 UTC m=+561.561601980" Jul 19 14:46:28.048419 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:28.048390 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:46:28.049803 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:28.049781 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:46:57.967837 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:57.967763 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk"] Jul 19 14:46:57.971004 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:57.970987 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:57.973559 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:57.973535 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 19 14:46:57.973688 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:57.973577 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 19 14:46:57.973688 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:57.973578 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nh8pg\"" Jul 19 14:46:57.980907 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:57.980854 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk"] Jul 19 14:46:58.010609 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.010575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvgg9\" (UniqueName: \"kubernetes.io/projected/b90b0b29-94c7-4369-933f-b88985ca6de9-kube-api-access-wvgg9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.010744 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.010628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.010744 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.010674 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.111550 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.111523 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.111550 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.111553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.111797 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.111603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvgg9\" (UniqueName: \"kubernetes.io/projected/b90b0b29-94c7-4369-933f-b88985ca6de9-kube-api-access-wvgg9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.111981 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.111958 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.112051 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.111968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.120586 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.120557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvgg9\" (UniqueName: \"kubernetes.io/projected/b90b0b29-94c7-4369-933f-b88985ca6de9-kube-api-access-wvgg9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.280613 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.280559 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:46:58.405790 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.405760 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk"] Jul 19 14:46:58.407803 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:46:58.407778 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90b0b29_94c7_4369_933f_b88985ca6de9.slice/crio-2008ad2136a8bc690977a25b49f10ff2ba9c05e21053e55ffcab8cdcfbc899d1 WatchSource:0}: Error finding container 2008ad2136a8bc690977a25b49f10ff2ba9c05e21053e55ffcab8cdcfbc899d1: Status 404 returned error can't find the container with id 2008ad2136a8bc690977a25b49f10ff2ba9c05e21053e55ffcab8cdcfbc899d1 Jul 19 14:46:58.439505 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.439483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" event={"ID":"b90b0b29-94c7-4369-933f-b88985ca6de9","Type":"ContainerStarted","Data":"2008ad2136a8bc690977a25b49f10ff2ba9c05e21053e55ffcab8cdcfbc899d1"} Jul 19 14:46:58.559660 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.559594 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs"] Jul 19 14:46:58.562940 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.562921 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.571911 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.571890 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs"] Jul 19 14:46:58.616197 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.616172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.616294 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.616277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.616336 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.616299 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45r2k\" (UniqueName: \"kubernetes.io/projected/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-kube-api-access-45r2k\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.717655 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.717629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.717746 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.717671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45r2k\" (UniqueName: \"kubernetes.io/projected/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-kube-api-access-45r2k\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.717746 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.717712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.717932 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.717912 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.718081 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.718058 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.725646 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.725627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45r2k\" (UniqueName: \"kubernetes.io/projected/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-kube-api-access-45r2k\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.872351 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.872322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:46:58.997593 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:58.997566 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs"] Jul 19 14:46:58.999759 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:46:58.999729 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b5f999_9e62_40a4_86d8_b8d2b745aa11.slice/crio-4d19e9a657f0111f03fe230f5c817101433cc3033099d79a00006c49255b7695 WatchSource:0}: Error finding container 4d19e9a657f0111f03fe230f5c817101433cc3033099d79a00006c49255b7695: Status 404 returned error can't find the container with id 4d19e9a657f0111f03fe230f5c817101433cc3033099d79a00006c49255b7695 Jul 19 14:46:59.156384 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.156353 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7"] Jul 19 14:46:59.159758 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.159744 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.167568 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.167546 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7"] Jul 19 14:46:59.222790 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.222755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.222933 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.222802 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.222933 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.222822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9z7m\" (UniqueName: \"kubernetes.io/projected/8949ef80-3fe1-4348-aebf-917365646e05-kube-api-access-t9z7m\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.323739 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.323713 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.323839 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.323756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.323839 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.323781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9z7m\" (UniqueName: \"kubernetes.io/projected/8949ef80-3fe1-4348-aebf-917365646e05-kube-api-access-t9z7m\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.324154 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.324133 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.324195 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.324144 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.332941 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.332919 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9z7m\" (UniqueName: \"kubernetes.io/projected/8949ef80-3fe1-4348-aebf-917365646e05-kube-api-access-t9z7m\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.445710 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.445654 2559 generic.go:356] "Generic (PLEG): container finished" podID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerID="6a62cacc8b666b678cd50694dfa2b8c9dcc490d08aaef7945980aed520a44184" exitCode=0 Jul 19 14:46:59.445795 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.445734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" event={"ID":"b90b0b29-94c7-4369-933f-b88985ca6de9","Type":"ContainerDied","Data":"6a62cacc8b666b678cd50694dfa2b8c9dcc490d08aaef7945980aed520a44184"} Jul 19 14:46:59.446219 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.446203 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 14:46:59.447285 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.447263 2559 generic.go:356] "Generic (PLEG): container finished" podID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerID="c363d229901a00c297dae32e4be4466b8fd1db4bbde87112fdbc7f1b5a7c0c9c" exitCode=0 Jul 19 14:46:59.447383 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.447347 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" event={"ID":"d1b5f999-9e62-40a4-86d8-b8d2b745aa11","Type":"ContainerDied","Data":"c363d229901a00c297dae32e4be4466b8fd1db4bbde87112fdbc7f1b5a7c0c9c"} Jul 19 14:46:59.447450 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.447383 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" event={"ID":"d1b5f999-9e62-40a4-86d8-b8d2b745aa11","Type":"ContainerStarted","Data":"4d19e9a657f0111f03fe230f5c817101433cc3033099d79a00006c49255b7695"} Jul 19 14:46:59.468809 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.468783 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:46:59.602104 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.602065 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7"] Jul 19 14:46:59.604668 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:46:59.604637 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8949ef80_3fe1_4348_aebf_917365646e05.slice/crio-858b5abc3321aaa73e86a16398b9af15a162d1e78bc9c61de8f40979ed64f0d9 WatchSource:0}: Error finding container 858b5abc3321aaa73e86a16398b9af15a162d1e78bc9c61de8f40979ed64f0d9: Status 404 returned error can't find the container with id 858b5abc3321aaa73e86a16398b9af15a162d1e78bc9c61de8f40979ed64f0d9 Jul 19 14:46:59.767480 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.767450 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8"] Jul 19 14:46:59.771004 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.770985 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:46:59.779840 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.779818 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8"] Jul 19 14:46:59.828958 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.828935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbfjv\" (UniqueName: \"kubernetes.io/projected/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-kube-api-access-qbfjv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:46:59.829059 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.828968 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:46:59.829115 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.829092 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:46:59.930108 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.930081 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:46:59.930223 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.930171 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:46:59.930223 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.930218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbfjv\" (UniqueName: \"kubernetes.io/projected/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-kube-api-access-qbfjv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:46:59.930452 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.930433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:46:59.930494 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.930466 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:46:59.943362 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:46:59.943337 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbfjv\" (UniqueName: \"kubernetes.io/projected/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-kube-api-access-qbfjv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:47:00.079833 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.079810 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:47:00.220687 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.220662 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8"] Jul 19 14:47:00.276744 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:47:00.276712 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac0d3b7_36d7_4d3d_bc4c_863bebdb2842.slice/crio-08a276e372bf8d24650fff39ad596412fba3fa39c791b39a151f8329cc2e8200 WatchSource:0}: Error finding container 08a276e372bf8d24650fff39ad596412fba3fa39c791b39a151f8329cc2e8200: Status 404 returned error can't find the container with id 08a276e372bf8d24650fff39ad596412fba3fa39c791b39a151f8329cc2e8200 Jul 19 14:47:00.453946 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.453920 2559 generic.go:356] "Generic (PLEG): container finished" podID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerID="3236355dc5bbc2f921440c49dbaa73c0db437626fbb38d6041e319e6d75b6a28" exitCode=0 Jul 19 14:47:00.454111 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.454003 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" event={"ID":"b90b0b29-94c7-4369-933f-b88985ca6de9","Type":"ContainerDied","Data":"3236355dc5bbc2f921440c49dbaa73c0db437626fbb38d6041e319e6d75b6a28"} Jul 19 14:47:00.455908 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.455860 2559 generic.go:356] "Generic (PLEG): container finished" podID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerID="e74c0174a942f224cc470d2e1ac58db47aa07b8790cbd3c108338d38121ef8ad" exitCode=0 Jul 19 14:47:00.456015 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.455889 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" event={"ID":"d1b5f999-9e62-40a4-86d8-b8d2b745aa11","Type":"ContainerDied","Data":"e74c0174a942f224cc470d2e1ac58db47aa07b8790cbd3c108338d38121ef8ad"} Jul 19 14:47:00.457580 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.457526 2559 generic.go:356] "Generic (PLEG): container finished" podID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerID="f67bd7e066b45dbc307661fa2763671aa94c1bb0596441fec912e6694cc4b1a5" exitCode=0 Jul 19 14:47:00.457666 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.457628 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" event={"ID":"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842","Type":"ContainerDied","Data":"f67bd7e066b45dbc307661fa2763671aa94c1bb0596441fec912e6694cc4b1a5"} Jul 19 14:47:00.457722 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.457695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" event={"ID":"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842","Type":"ContainerStarted","Data":"08a276e372bf8d24650fff39ad596412fba3fa39c791b39a151f8329cc2e8200"} Jul 19 14:47:00.459173 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.459150 2559 generic.go:356] "Generic (PLEG): container finished" podID="8949ef80-3fe1-4348-aebf-917365646e05" containerID="c40a0bfcf8c019c2fea5175b95d611c7968335f9d3294a7a2371a7adf18bb1a2" exitCode=0 Jul 19 14:47:00.459254 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.459220 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" event={"ID":"8949ef80-3fe1-4348-aebf-917365646e05","Type":"ContainerDied","Data":"c40a0bfcf8c019c2fea5175b95d611c7968335f9d3294a7a2371a7adf18bb1a2"} Jul 19 14:47:00.459391 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:00.459281 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" event={"ID":"8949ef80-3fe1-4348-aebf-917365646e05","Type":"ContainerStarted","Data":"858b5abc3321aaa73e86a16398b9af15a162d1e78bc9c61de8f40979ed64f0d9"} Jul 19 14:47:01.465811 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:01.465782 2559 generic.go:356] "Generic (PLEG): container finished" podID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerID="ffab55cc3633b2206189d5d4b464b443fcbc969d7cc386822eb15c2e7a25800f" exitCode=0 Jul 19 14:47:01.466215 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:01.465876 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" event={"ID":"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842","Type":"ContainerDied","Data":"ffab55cc3633b2206189d5d4b464b443fcbc969d7cc386822eb15c2e7a25800f"} Jul 19 14:47:01.468075 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:01.468051 2559 generic.go:356] "Generic (PLEG): container finished" podID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerID="b48a5ec0f5766464aa630a890ef36cfb9bbaa345153ff96497ded82ac8939472" exitCode=0 Jul 19 14:47:01.468167 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:01.468135 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" event={"ID":"b90b0b29-94c7-4369-933f-b88985ca6de9","Type":"ContainerDied","Data":"b48a5ec0f5766464aa630a890ef36cfb9bbaa345153ff96497ded82ac8939472"} Jul 19 14:47:01.470168 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:01.470146 2559 generic.go:356] "Generic (PLEG): container finished" podID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerID="d69a688adf5e4050e78e614b4e0a4a8da6ece2f24379bed16927cd4cc7c3a4ef" exitCode=0 Jul 19 14:47:01.470253 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:01.470193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" event={"ID":"d1b5f999-9e62-40a4-86d8-b8d2b745aa11","Type":"ContainerDied","Data":"d69a688adf5e4050e78e614b4e0a4a8da6ece2f24379bed16927cd4cc7c3a4ef"} Jul 19 14:47:02.477121 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.477041 2559 generic.go:356] "Generic (PLEG): container finished" podID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerID="1d38028a704087fd9c0a73cf01a84a2a646eb7eac3c6ba197de1915b0254fc7b" exitCode=0 Jul 19 14:47:02.477505 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.477129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" event={"ID":"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842","Type":"ContainerDied","Data":"1d38028a704087fd9c0a73cf01a84a2a646eb7eac3c6ba197de1915b0254fc7b"} Jul 19 14:47:02.478830 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.478803 2559 generic.go:356] "Generic (PLEG): container finished" podID="8949ef80-3fe1-4348-aebf-917365646e05" containerID="4f72c963d73d0531baefc198be2aefb3e22dfc54b8047841bbd78b0cac55fb65" exitCode=0 Jul 19 14:47:02.478972 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.478893 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" event={"ID":"8949ef80-3fe1-4348-aebf-917365646e05","Type":"ContainerDied","Data":"4f72c963d73d0531baefc198be2aefb3e22dfc54b8047841bbd78b0cac55fb65"} Jul 19 14:47:02.692950 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.692925 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:47:02.695956 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.695929 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:47:02.757315 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.757248 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-util\") pod \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " Jul 19 14:47:02.757315 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.757310 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45r2k\" (UniqueName: \"kubernetes.io/projected/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-kube-api-access-45r2k\") pod \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " Jul 19 14:47:02.757449 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.757353 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-bundle\") pod \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\" (UID: \"d1b5f999-9e62-40a4-86d8-b8d2b745aa11\") " Jul 19 14:47:02.757449 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.757378 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-bundle\") pod \"b90b0b29-94c7-4369-933f-b88985ca6de9\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " Jul 19 14:47:02.757535 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.757520 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-util\") pod \"b90b0b29-94c7-4369-933f-b88985ca6de9\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " Jul 19 14:47:02.757604 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.757591 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvgg9\" (UniqueName: \"kubernetes.io/projected/b90b0b29-94c7-4369-933f-b88985ca6de9-kube-api-access-wvgg9\") pod \"b90b0b29-94c7-4369-933f-b88985ca6de9\" (UID: \"b90b0b29-94c7-4369-933f-b88985ca6de9\") " Jul 19 14:47:02.758005 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.757976 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-bundle" (OuterVolumeSpecName: "bundle") pod "b90b0b29-94c7-4369-933f-b88985ca6de9" (UID: "b90b0b29-94c7-4369-933f-b88985ca6de9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:47:02.758356 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.758332 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-bundle" (OuterVolumeSpecName: "bundle") pod "d1b5f999-9e62-40a4-86d8-b8d2b745aa11" (UID: "d1b5f999-9e62-40a4-86d8-b8d2b745aa11"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:47:02.760133 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.760111 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-kube-api-access-45r2k" (OuterVolumeSpecName: "kube-api-access-45r2k") pod "d1b5f999-9e62-40a4-86d8-b8d2b745aa11" (UID: "d1b5f999-9e62-40a4-86d8-b8d2b745aa11"). InnerVolumeSpecName "kube-api-access-45r2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:47:02.760252 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.760141 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90b0b29-94c7-4369-933f-b88985ca6de9-kube-api-access-wvgg9" (OuterVolumeSpecName: "kube-api-access-wvgg9") pod "b90b0b29-94c7-4369-933f-b88985ca6de9" (UID: "b90b0b29-94c7-4369-933f-b88985ca6de9"). InnerVolumeSpecName "kube-api-access-wvgg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:47:02.762833 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.762812 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-util" (OuterVolumeSpecName: "util") pod "b90b0b29-94c7-4369-933f-b88985ca6de9" (UID: "b90b0b29-94c7-4369-933f-b88985ca6de9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:47:02.762917 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.762855 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-util" (OuterVolumeSpecName: "util") pod "d1b5f999-9e62-40a4-86d8-b8d2b745aa11" (UID: "d1b5f999-9e62-40a4-86d8-b8d2b745aa11"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:47:02.858452 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.858423 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:02.858452 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.858449 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45r2k\" (UniqueName: \"kubernetes.io/projected/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-kube-api-access-45r2k\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:02.858452 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.858460 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1b5f999-9e62-40a4-86d8-b8d2b745aa11-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:02.858598 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.858469 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:02.858598 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.858476 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b90b0b29-94c7-4369-933f-b88985ca6de9-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:02.858598 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:02.858484 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvgg9\" (UniqueName: \"kubernetes.io/projected/b90b0b29-94c7-4369-933f-b88985ca6de9-kube-api-access-wvgg9\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:03.485918 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.485810 2559 generic.go:356] "Generic (PLEG): container finished" podID="8949ef80-3fe1-4348-aebf-917365646e05" containerID="64044caa56e8ec4482b4c65d93d63573175ea127e508935a9ee05d1f1fdbfb3b" exitCode=0 Jul 19 14:47:03.486364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.485909 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" event={"ID":"8949ef80-3fe1-4348-aebf-917365646e05","Type":"ContainerDied","Data":"64044caa56e8ec4482b4c65d93d63573175ea127e508935a9ee05d1f1fdbfb3b"} Jul 19 14:47:03.494006 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.493971 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" event={"ID":"b90b0b29-94c7-4369-933f-b88985ca6de9","Type":"ContainerDied","Data":"2008ad2136a8bc690977a25b49f10ff2ba9c05e21053e55ffcab8cdcfbc899d1"} Jul 19 14:47:03.494211 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.494046 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2008ad2136a8bc690977a25b49f10ff2ba9c05e21053e55ffcab8cdcfbc899d1" Jul 19 14:47:03.494211 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.493985 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk" Jul 19 14:47:03.495759 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.495741 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" Jul 19 14:47:03.495848 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.495769 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs" event={"ID":"d1b5f999-9e62-40a4-86d8-b8d2b745aa11","Type":"ContainerDied","Data":"4d19e9a657f0111f03fe230f5c817101433cc3033099d79a00006c49255b7695"} Jul 19 14:47:03.495848 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.495806 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d19e9a657f0111f03fe230f5c817101433cc3033099d79a00006c49255b7695" Jul 19 14:47:03.630706 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.630683 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:47:03.765595 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.765524 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbfjv\" (UniqueName: \"kubernetes.io/projected/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-kube-api-access-qbfjv\") pod \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " Jul 19 14:47:03.765730 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.765603 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-bundle\") pod \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " Jul 19 14:47:03.765730 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.765641 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-util\") pod \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\" (UID: \"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842\") " Jul 19 14:47:03.766374 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.766347 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-bundle" (OuterVolumeSpecName: "bundle") pod "3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" (UID: "3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:47:03.767663 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.767641 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-kube-api-access-qbfjv" (OuterVolumeSpecName: "kube-api-access-qbfjv") pod "3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" (UID: "3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842"). InnerVolumeSpecName "kube-api-access-qbfjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:47:03.771105 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.771070 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-util" (OuterVolumeSpecName: "util") pod "3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" (UID: "3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:47:03.866819 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.866792 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbfjv\" (UniqueName: \"kubernetes.io/projected/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-kube-api-access-qbfjv\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:03.866819 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.866817 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:03.866984 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:03.866827 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:04.504429 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.504357 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" Jul 19 14:47:04.504429 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.504388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8" event={"ID":"3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842","Type":"ContainerDied","Data":"08a276e372bf8d24650fff39ad596412fba3fa39c791b39a151f8329cc2e8200"} Jul 19 14:47:04.504771 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.504436 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08a276e372bf8d24650fff39ad596412fba3fa39c791b39a151f8329cc2e8200" Jul 19 14:47:04.628325 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.628304 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:47:04.773767 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.773688 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-util\") pod \"8949ef80-3fe1-4348-aebf-917365646e05\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " Jul 19 14:47:04.773944 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.773781 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-bundle\") pod \"8949ef80-3fe1-4348-aebf-917365646e05\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " Jul 19 14:47:04.773944 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.773810 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9z7m\" (UniqueName: \"kubernetes.io/projected/8949ef80-3fe1-4348-aebf-917365646e05-kube-api-access-t9z7m\") pod \"8949ef80-3fe1-4348-aebf-917365646e05\" (UID: \"8949ef80-3fe1-4348-aebf-917365646e05\") " Jul 19 14:47:04.774241 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.774210 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-bundle" (OuterVolumeSpecName: "bundle") pod "8949ef80-3fe1-4348-aebf-917365646e05" (UID: "8949ef80-3fe1-4348-aebf-917365646e05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:47:04.775856 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.775837 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8949ef80-3fe1-4348-aebf-917365646e05-kube-api-access-t9z7m" (OuterVolumeSpecName: "kube-api-access-t9z7m") pod "8949ef80-3fe1-4348-aebf-917365646e05" (UID: "8949ef80-3fe1-4348-aebf-917365646e05"). InnerVolumeSpecName "kube-api-access-t9z7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:47:04.821273 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.821248 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-util" (OuterVolumeSpecName: "util") pod "8949ef80-3fe1-4348-aebf-917365646e05" (UID: "8949ef80-3fe1-4348-aebf-917365646e05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:47:04.874372 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.874331 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:04.874372 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.874369 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9z7m\" (UniqueName: \"kubernetes.io/projected/8949ef80-3fe1-4348-aebf-917365646e05-kube-api-access-t9z7m\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:04.874372 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:04.874380 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8949ef80-3fe1-4348-aebf-917365646e05-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:47:05.510634 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:05.510604 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" Jul 19 14:47:05.511048 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:05.510643 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7" event={"ID":"8949ef80-3fe1-4348-aebf-917365646e05","Type":"ContainerDied","Data":"858b5abc3321aaa73e86a16398b9af15a162d1e78bc9c61de8f40979ed64f0d9"} Jul 19 14:47:05.511048 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:05.510690 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858b5abc3321aaa73e86a16398b9af15a162d1e78bc9c61de8f40979ed64f0d9" Jul 19 14:47:25.476720 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.476685 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62"] Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477044 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerName="util" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477058 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerName="util" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477066 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerName="extract" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477071 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerName="extract" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477079 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerName="extract" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477086 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerName="extract" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477094 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8949ef80-3fe1-4348-aebf-917365646e05" containerName="extract" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477099 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8949ef80-3fe1-4348-aebf-917365646e05" containerName="extract" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477107 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerName="util" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477113 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerName="util" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477120 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerName="util" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477125 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerName="util" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477135 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerName="pull" Jul 19 14:47:25.477128 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477140 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerName="pull" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477147 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerName="pull" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477152 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerName="pull" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477159 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerName="extract" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477164 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerName="extract" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477175 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8949ef80-3fe1-4348-aebf-917365646e05" containerName="util" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477180 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8949ef80-3fe1-4348-aebf-917365646e05" containerName="util" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477185 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8949ef80-3fe1-4348-aebf-917365646e05" containerName="pull" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477190 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8949ef80-3fe1-4348-aebf-917365646e05" containerName="pull" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477197 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerName="pull" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477203 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerName="pull" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477259 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b90b0b29-94c7-4369-933f-b88985ca6de9" containerName="extract" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477267 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842" containerName="extract" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477275 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b5f999-9e62-40a4-86d8-b8d2b745aa11" containerName="extract" Jul 19 14:47:25.477530 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.477281 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8949ef80-3fe1-4348-aebf-917365646e05" containerName="extract" Jul 19 14:47:25.480316 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.480297 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.482739 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.482713 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 19 14:47:25.482831 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.482741 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 19 14:47:25.482831 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.482765 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 19 14:47:25.482940 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.482723 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nh8pg\"" Jul 19 14:47:25.483605 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.483590 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 19 14:47:25.490571 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.490553 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62"] Jul 19 14:47:25.556544 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.556515 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/509523fe-2db9-4213-9c7d-cbbf23101843-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jpm62\" (UID: \"509523fe-2db9-4213-9c7d-cbbf23101843\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.556660 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.556581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fnn\" (UniqueName: \"kubernetes.io/projected/509523fe-2db9-4213-9c7d-cbbf23101843-kube-api-access-n5fnn\") pod \"kuadrant-console-plugin-66b5774985-jpm62\" (UID: \"509523fe-2db9-4213-9c7d-cbbf23101843\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.556660 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.556599 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/509523fe-2db9-4213-9c7d-cbbf23101843-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jpm62\" (UID: \"509523fe-2db9-4213-9c7d-cbbf23101843\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.657289 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.657255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/509523fe-2db9-4213-9c7d-cbbf23101843-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jpm62\" (UID: \"509523fe-2db9-4213-9c7d-cbbf23101843\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.657457 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.657356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fnn\" (UniqueName: \"kubernetes.io/projected/509523fe-2db9-4213-9c7d-cbbf23101843-kube-api-access-n5fnn\") pod \"kuadrant-console-plugin-66b5774985-jpm62\" (UID: \"509523fe-2db9-4213-9c7d-cbbf23101843\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.657457 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.657384 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/509523fe-2db9-4213-9c7d-cbbf23101843-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jpm62\" (UID: \"509523fe-2db9-4213-9c7d-cbbf23101843\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.658046 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.658022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/509523fe-2db9-4213-9c7d-cbbf23101843-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jpm62\" (UID: \"509523fe-2db9-4213-9c7d-cbbf23101843\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.659638 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.659619 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/509523fe-2db9-4213-9c7d-cbbf23101843-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jpm62\" (UID: \"509523fe-2db9-4213-9c7d-cbbf23101843\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.664508 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.664490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fnn\" (UniqueName: \"kubernetes.io/projected/509523fe-2db9-4213-9c7d-cbbf23101843-kube-api-access-n5fnn\") pod \"kuadrant-console-plugin-66b5774985-jpm62\" (UID: \"509523fe-2db9-4213-9c7d-cbbf23101843\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.790702 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.790619 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" Jul 19 14:47:25.922833 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:25.922792 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62"] Jul 19 14:47:25.924730 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:47:25.924704 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod509523fe_2db9_4213_9c7d_cbbf23101843.slice/crio-510299ec3831a56df1df98035c028b140c4283f8b20b46550541220375dd0f10 WatchSource:0}: Error finding container 510299ec3831a56df1df98035c028b140c4283f8b20b46550541220375dd0f10: Status 404 returned error can't find the container with id 510299ec3831a56df1df98035c028b140c4283f8b20b46550541220375dd0f10 Jul 19 14:47:26.616641 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:26.616605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" event={"ID":"509523fe-2db9-4213-9c7d-cbbf23101843","Type":"ContainerStarted","Data":"510299ec3831a56df1df98035c028b140c4283f8b20b46550541220375dd0f10"} Jul 19 14:47:47.740743 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:47.740709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" event={"ID":"509523fe-2db9-4213-9c7d-cbbf23101843","Type":"ContainerStarted","Data":"c985603280ebf8f5dd6c715a72781787ee25d02ad9fb5861beb5215ff6c52d6f"} Jul 19 14:47:47.756720 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:47:47.756674 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jpm62" podStartSLOduration=1.3865504880000001 podStartE2EDuration="22.756660538s" totalImagesPullingTime="21.37011005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:47:47.754758194 +0000 UTC m=+680.176982519" watchObservedRunningTime="2026-07-19 14:47:47.756660538 +0000 UTC m=+680.178884867" Jul 19 14:48:10.954365 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:10.954330 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:48:10.990572 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:10.990540 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:48:10.990572 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:10.990579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:48:10.990756 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:10.990661 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:10.993062 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:10.993043 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 19 14:48:11.139427 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.139393 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hsnx\" (UniqueName: \"kubernetes.io/projected/c1f07116-eaec-412e-8e8f-bb9c806a03eb-kube-api-access-2hsnx\") pod \"limitador-limitador-54d6d988d5-8x9kp\" (UID: \"c1f07116-eaec-412e-8e8f-bb9c806a03eb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:11.139427 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.139439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c1f07116-eaec-412e-8e8f-bb9c806a03eb-config-file\") pod \"limitador-limitador-54d6d988d5-8x9kp\" (UID: \"c1f07116-eaec-412e-8e8f-bb9c806a03eb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:11.240775 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.240695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsnx\" (UniqueName: \"kubernetes.io/projected/c1f07116-eaec-412e-8e8f-bb9c806a03eb-kube-api-access-2hsnx\") pod \"limitador-limitador-54d6d988d5-8x9kp\" (UID: \"c1f07116-eaec-412e-8e8f-bb9c806a03eb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:11.240775 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.240736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c1f07116-eaec-412e-8e8f-bb9c806a03eb-config-file\") pod \"limitador-limitador-54d6d988d5-8x9kp\" (UID: \"c1f07116-eaec-412e-8e8f-bb9c806a03eb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:11.241364 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.241344 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c1f07116-eaec-412e-8e8f-bb9c806a03eb-config-file\") pod \"limitador-limitador-54d6d988d5-8x9kp\" (UID: \"c1f07116-eaec-412e-8e8f-bb9c806a03eb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:11.247611 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.247588 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-b4d7g"] Jul 19 14:48:11.250655 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.250632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsnx\" (UniqueName: \"kubernetes.io/projected/c1f07116-eaec-412e-8e8f-bb9c806a03eb-kube-api-access-2hsnx\") pod \"limitador-limitador-54d6d988d5-8x9kp\" (UID: \"c1f07116-eaec-412e-8e8f-bb9c806a03eb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:11.291984 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.291961 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-b4d7g"] Jul 19 14:48:11.292088 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.292045 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b4d7g" Jul 19 14:48:11.294475 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.294454 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k9sdp\"" Jul 19 14:48:11.300252 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.300232 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:11.430231 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.430208 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:48:11.432252 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:48:11.432215 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f07116_eaec_412e_8e8f_bb9c806a03eb.slice/crio-8c6725a1e72cd0ab6ef681cd0807ad29a8a72056d8d44d4eb4ea04bf5eb51556 WatchSource:0}: Error finding container 8c6725a1e72cd0ab6ef681cd0807ad29a8a72056d8d44d4eb4ea04bf5eb51556: Status 404 returned error can't find the container with id 8c6725a1e72cd0ab6ef681cd0807ad29a8a72056d8d44d4eb4ea04bf5eb51556 Jul 19 14:48:11.442197 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.442172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb2hs\" (UniqueName: \"kubernetes.io/projected/e97de343-c13c-4da8-ae16-bfe26b02443d-kube-api-access-hb2hs\") pod \"authorino-58cd955675-b4d7g\" (UID: \"e97de343-c13c-4da8-ae16-bfe26b02443d\") " pod="kuadrant-system/authorino-58cd955675-b4d7g" Jul 19 14:48:11.543296 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.543221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2hs\" (UniqueName: \"kubernetes.io/projected/e97de343-c13c-4da8-ae16-bfe26b02443d-kube-api-access-hb2hs\") pod \"authorino-58cd955675-b4d7g\" (UID: \"e97de343-c13c-4da8-ae16-bfe26b02443d\") " pod="kuadrant-system/authorino-58cd955675-b4d7g" Jul 19 14:48:11.550702 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.550677 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2hs\" (UniqueName: \"kubernetes.io/projected/e97de343-c13c-4da8-ae16-bfe26b02443d-kube-api-access-hb2hs\") pod \"authorino-58cd955675-b4d7g\" (UID: \"e97de343-c13c-4da8-ae16-bfe26b02443d\") " pod="kuadrant-system/authorino-58cd955675-b4d7g" Jul 19 14:48:11.601148 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.601123 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b4d7g" Jul 19 14:48:11.726894 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.726686 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-b4d7g"] Jul 19 14:48:11.729345 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:48:11.729295 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode97de343_c13c_4da8_ae16_bfe26b02443d.slice/crio-71121f54dd9a206952b48a9b087fdf998d3465809560a3e6cf2a4c066f89b282 WatchSource:0}: Error finding container 71121f54dd9a206952b48a9b087fdf998d3465809560a3e6cf2a4c066f89b282: Status 404 returned error can't find the container with id 71121f54dd9a206952b48a9b087fdf998d3465809560a3e6cf2a4c066f89b282 Jul 19 14:48:11.865019 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.864983 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" event={"ID":"c1f07116-eaec-412e-8e8f-bb9c806a03eb","Type":"ContainerStarted","Data":"8c6725a1e72cd0ab6ef681cd0807ad29a8a72056d8d44d4eb4ea04bf5eb51556"} Jul 19 14:48:11.866691 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:11.866659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b4d7g" event={"ID":"e97de343-c13c-4da8-ae16-bfe26b02443d","Type":"ContainerStarted","Data":"71121f54dd9a206952b48a9b087fdf998d3465809560a3e6cf2a4c066f89b282"} Jul 19 14:48:15.892742 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:15.892701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" event={"ID":"c1f07116-eaec-412e-8e8f-bb9c806a03eb","Type":"ContainerStarted","Data":"eab4905c439b93be419d625cca6d3e0a711ef6741ab9432a7ec94752f47d6920"} Jul 19 14:48:15.893351 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:15.892886 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:15.894217 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:15.894192 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b4d7g" event={"ID":"e97de343-c13c-4da8-ae16-bfe26b02443d","Type":"ContainerStarted","Data":"14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b"} Jul 19 14:48:15.911111 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:15.911029 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" podStartSLOduration=1.697157073 podStartE2EDuration="5.911015638s" totalImagesPullingTime="4.213858565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:48:15.908147238 +0000 UTC m=+708.330371568" watchObservedRunningTime="2026-07-19 14:48:15.911015638 +0000 UTC m=+708.333239970" Jul 19 14:48:15.926240 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:15.926197 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-b4d7g" podStartSLOduration=1.00766687 podStartE2EDuration="4.926182817s" totalImagesPullingTime="3.918515947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:48:15.925267919 +0000 UTC m=+708.347492249" watchObservedRunningTime="2026-07-19 14:48:15.926182817 +0000 UTC m=+708.348407148" Jul 19 14:48:16.433690 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:16.433658 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-b4d7g"] Jul 19 14:48:17.904415 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:17.904358 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-b4d7g" podUID="e97de343-c13c-4da8-ae16-bfe26b02443d" containerName="authorino" containerID="cri-o://14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b" gracePeriod=30 Jul 19 14:48:18.187803 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.187776 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b4d7g" Jul 19 14:48:18.303395 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.303363 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb2hs\" (UniqueName: \"kubernetes.io/projected/e97de343-c13c-4da8-ae16-bfe26b02443d-kube-api-access-hb2hs\") pod \"e97de343-c13c-4da8-ae16-bfe26b02443d\" (UID: \"e97de343-c13c-4da8-ae16-bfe26b02443d\") " Jul 19 14:48:18.305352 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.305330 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97de343-c13c-4da8-ae16-bfe26b02443d-kube-api-access-hb2hs" (OuterVolumeSpecName: "kube-api-access-hb2hs") pod "e97de343-c13c-4da8-ae16-bfe26b02443d" (UID: "e97de343-c13c-4da8-ae16-bfe26b02443d"). InnerVolumeSpecName "kube-api-access-hb2hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:48:18.404008 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.403975 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb2hs\" (UniqueName: \"kubernetes.io/projected/e97de343-c13c-4da8-ae16-bfe26b02443d-kube-api-access-hb2hs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:48:18.910670 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.910637 2559 generic.go:356] "Generic (PLEG): container finished" podID="e97de343-c13c-4da8-ae16-bfe26b02443d" containerID="14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b" exitCode=0 Jul 19 14:48:18.911180 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.910680 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b4d7g" Jul 19 14:48:18.911180 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.910720 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b4d7g" event={"ID":"e97de343-c13c-4da8-ae16-bfe26b02443d","Type":"ContainerDied","Data":"14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b"} Jul 19 14:48:18.911180 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.910763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b4d7g" event={"ID":"e97de343-c13c-4da8-ae16-bfe26b02443d","Type":"ContainerDied","Data":"71121f54dd9a206952b48a9b087fdf998d3465809560a3e6cf2a4c066f89b282"} Jul 19 14:48:18.911180 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.910783 2559 scope.go:117] "RemoveContainer" containerID="14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b" Jul 19 14:48:18.919878 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.919852 2559 scope.go:117] "RemoveContainer" containerID="14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b" Jul 19 14:48:18.920141 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:48:18.920125 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b\": container with ID starting with 14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b not found: ID does not exist" containerID="14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b" Jul 19 14:48:18.920200 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.920154 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b"} err="failed to get container status \"14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b\": rpc error: code = NotFound desc = could not find container \"14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b\": container with ID starting with 14a3fb5d62f1b203a1713492eada9a769c88938c94b9203454a261318b86ec5b not found: ID does not exist" Jul 19 14:48:18.932100 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.932077 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-b4d7g"] Jul 19 14:48:18.935645 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:18.935625 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-b4d7g"] Jul 19 14:48:20.106653 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:20.106618 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e97de343-c13c-4da8-ae16-bfe26b02443d" path="/var/lib/kubelet/pods/e97de343-c13c-4da8-ae16-bfe26b02443d/volumes" Jul 19 14:48:26.900062 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:26.899980 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8x9kp" Jul 19 14:48:44.119588 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.119555 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tw84w"] Jul 19 14:48:44.120063 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.119854 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97de343-c13c-4da8-ae16-bfe26b02443d" containerName="authorino" Jul 19 14:48:44.120063 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.119878 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97de343-c13c-4da8-ae16-bfe26b02443d" containerName="authorino" Jul 19 14:48:44.120063 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.119949 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e97de343-c13c-4da8-ae16-bfe26b02443d" containerName="authorino" Jul 19 14:48:44.126618 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.126599 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tw84w" Jul 19 14:48:44.128998 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.128975 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k9sdp\"" Jul 19 14:48:44.130481 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.130448 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tw84w"] Jul 19 14:48:44.224194 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.224165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tm5p\" (UniqueName: \"kubernetes.io/projected/0c2e5720-61ad-446b-9282-9d14f5415c0c-kube-api-access-5tm5p\") pod \"authorino-5d4b67c446-tw84w\" (UID: \"0c2e5720-61ad-446b-9282-9d14f5415c0c\") " pod="kuadrant-system/authorino-5d4b67c446-tw84w" Jul 19 14:48:44.325159 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.325117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tm5p\" (UniqueName: \"kubernetes.io/projected/0c2e5720-61ad-446b-9282-9d14f5415c0c-kube-api-access-5tm5p\") pod \"authorino-5d4b67c446-tw84w\" (UID: \"0c2e5720-61ad-446b-9282-9d14f5415c0c\") " pod="kuadrant-system/authorino-5d4b67c446-tw84w" Jul 19 14:48:44.333008 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.332978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tm5p\" (UniqueName: \"kubernetes.io/projected/0c2e5720-61ad-446b-9282-9d14f5415c0c-kube-api-access-5tm5p\") pod \"authorino-5d4b67c446-tw84w\" (UID: \"0c2e5720-61ad-446b-9282-9d14f5415c0c\") " pod="kuadrant-system/authorino-5d4b67c446-tw84w" Jul 19 14:48:44.371509 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.371436 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tw84w"] Jul 19 14:48:44.371683 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.371671 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tw84w" Jul 19 14:48:44.396556 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.396530 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79465fcc47-kj62s"] Jul 19 14:48:44.403046 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.403011 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79465fcc47-kj62s" Jul 19 14:48:44.405517 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.405484 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79465fcc47-kj62s"] Jul 19 14:48:44.424789 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.424763 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79465fcc47-kj62s"] Jul 19 14:48:44.425049 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:48:44.425029 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xvsrk], unattached volumes=[], failed to process volumes=[kube-api-access-xvsrk]: context canceled" pod="kuadrant-system/authorino-79465fcc47-kj62s" podUID="2c5026e8-b79a-4625-b60c-1749bc19cacd" Jul 19 14:48:44.510826 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.510798 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tw84w"] Jul 19 14:48:44.512555 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:48:44.512532 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c2e5720_61ad_446b_9282_9d14f5415c0c.slice/crio-82c882bad729a02d073ab49d71c9d4e4fc0ed7ee146a696b93aad94b692c08b8 WatchSource:0}: Error finding container 82c882bad729a02d073ab49d71c9d4e4fc0ed7ee146a696b93aad94b692c08b8: Status 404 returned error can't find the container with id 82c882bad729a02d073ab49d71c9d4e4fc0ed7ee146a696b93aad94b692c08b8 Jul 19 14:48:44.526614 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.526581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvsrk\" (UniqueName: \"kubernetes.io/projected/2c5026e8-b79a-4625-b60c-1749bc19cacd-kube-api-access-xvsrk\") pod \"authorino-79465fcc47-kj62s\" (UID: \"2c5026e8-b79a-4625-b60c-1749bc19cacd\") " pod="kuadrant-system/authorino-79465fcc47-kj62s" Jul 19 14:48:44.627904 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.627813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvsrk\" (UniqueName: \"kubernetes.io/projected/2c5026e8-b79a-4625-b60c-1749bc19cacd-kube-api-access-xvsrk\") pod \"authorino-79465fcc47-kj62s\" (UID: \"2c5026e8-b79a-4625-b60c-1749bc19cacd\") " pod="kuadrant-system/authorino-79465fcc47-kj62s" Jul 19 14:48:44.635160 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:44.635138 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvsrk\" (UniqueName: \"kubernetes.io/projected/2c5026e8-b79a-4625-b60c-1749bc19cacd-kube-api-access-xvsrk\") pod \"authorino-79465fcc47-kj62s\" (UID: \"2c5026e8-b79a-4625-b60c-1749bc19cacd\") " pod="kuadrant-system/authorino-79465fcc47-kj62s" Jul 19 14:48:45.051097 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:45.051028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tw84w" event={"ID":"0c2e5720-61ad-446b-9282-9d14f5415c0c","Type":"ContainerStarted","Data":"82c882bad729a02d073ab49d71c9d4e4fc0ed7ee146a696b93aad94b692c08b8"} Jul 19 14:48:45.051097 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:45.051037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79465fcc47-kj62s" Jul 19 14:48:45.069347 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:45.069326 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79465fcc47-kj62s" Jul 19 14:48:45.131975 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:45.131958 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvsrk\" (UniqueName: \"kubernetes.io/projected/2c5026e8-b79a-4625-b60c-1749bc19cacd-kube-api-access-xvsrk\") pod \"2c5026e8-b79a-4625-b60c-1749bc19cacd\" (UID: \"2c5026e8-b79a-4625-b60c-1749bc19cacd\") " Jul 19 14:48:45.133846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:45.133811 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5026e8-b79a-4625-b60c-1749bc19cacd-kube-api-access-xvsrk" (OuterVolumeSpecName: "kube-api-access-xvsrk") pod "2c5026e8-b79a-4625-b60c-1749bc19cacd" (UID: "2c5026e8-b79a-4625-b60c-1749bc19cacd"). InnerVolumeSpecName "kube-api-access-xvsrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:48:45.232666 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:45.232632 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvsrk\" (UniqueName: \"kubernetes.io/projected/2c5026e8-b79a-4625-b60c-1749bc19cacd-kube-api-access-xvsrk\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:48:46.057887 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.057840 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79465fcc47-kj62s" Jul 19 14:48:46.058110 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.057900 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tw84w" event={"ID":"0c2e5720-61ad-446b-9282-9d14f5415c0c","Type":"ContainerStarted","Data":"b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf"} Jul 19 14:48:46.058110 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.057922 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-tw84w" podUID="0c2e5720-61ad-446b-9282-9d14f5415c0c" containerName="authorino" containerID="cri-o://b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf" gracePeriod=30 Jul 19 14:48:46.076038 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.075984 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-tw84w" podStartSLOduration=1.525372483 podStartE2EDuration="2.075968563s" totalImagesPullingTime="550.59608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:48:46.073987915 +0000 UTC m=+738.496212250" watchObservedRunningTime="2026-07-19 14:48:46.075968563 +0000 UTC m=+738.498192893" Jul 19 14:48:46.097983 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.097940 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79465fcc47-kj62s"] Jul 19 14:48:46.099966 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.099943 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79465fcc47-kj62s"] Jul 19 14:48:46.107541 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.107513 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c5026e8-b79a-4625-b60c-1749bc19cacd" path="/var/lib/kubelet/pods/2c5026e8-b79a-4625-b60c-1749bc19cacd/volumes" Jul 19 14:48:46.338645 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.338619 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tw84w" Jul 19 14:48:46.443179 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.443146 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tm5p\" (UniqueName: \"kubernetes.io/projected/0c2e5720-61ad-446b-9282-9d14f5415c0c-kube-api-access-5tm5p\") pod \"0c2e5720-61ad-446b-9282-9d14f5415c0c\" (UID: \"0c2e5720-61ad-446b-9282-9d14f5415c0c\") " Jul 19 14:48:46.445145 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.445115 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2e5720-61ad-446b-9282-9d14f5415c0c-kube-api-access-5tm5p" (OuterVolumeSpecName: "kube-api-access-5tm5p") pod "0c2e5720-61ad-446b-9282-9d14f5415c0c" (UID: "0c2e5720-61ad-446b-9282-9d14f5415c0c"). InnerVolumeSpecName "kube-api-access-5tm5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:48:46.544338 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:46.544303 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tm5p\" (UniqueName: \"kubernetes.io/projected/0c2e5720-61ad-446b-9282-9d14f5415c0c-kube-api-access-5tm5p\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 14:48:47.063937 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:47.063897 2559 generic.go:356] "Generic (PLEG): container finished" podID="0c2e5720-61ad-446b-9282-9d14f5415c0c" containerID="b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf" exitCode=0 Jul 19 14:48:47.063937 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:47.063937 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tw84w" Jul 19 14:48:47.064151 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:47.063982 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tw84w" event={"ID":"0c2e5720-61ad-446b-9282-9d14f5415c0c","Type":"ContainerDied","Data":"b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf"} Jul 19 14:48:47.064151 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:47.064022 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tw84w" event={"ID":"0c2e5720-61ad-446b-9282-9d14f5415c0c","Type":"ContainerDied","Data":"82c882bad729a02d073ab49d71c9d4e4fc0ed7ee146a696b93aad94b692c08b8"} Jul 19 14:48:47.064151 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:47.064041 2559 scope.go:117] "RemoveContainer" containerID="b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf" Jul 19 14:48:47.073061 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:47.073047 2559 scope.go:117] "RemoveContainer" containerID="b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf" Jul 19 14:48:47.073309 ip-10-0-134-156 kubenswrapper[2559]: E0719 14:48:47.073292 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf\": container with ID starting with b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf not found: ID does not exist" containerID="b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf" Jul 19 14:48:47.073373 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:47.073321 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf"} err="failed to get container status \"b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf\": rpc error: code = NotFound desc = could not find container \"b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf\": container with ID starting with b8a3a508eac77b2e3ddd42c09de3400675b82ff36371c45816e2180a9c1d5cbf not found: ID does not exist" Jul 19 14:48:47.084159 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:47.084136 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tw84w"] Jul 19 14:48:47.087971 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:47.087952 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tw84w"] Jul 19 14:48:48.112382 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:48:48.112350 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2e5720-61ad-446b-9282-9d14f5415c0c" path="/var/lib/kubelet/pods/0c2e5720-61ad-446b-9282-9d14f5415c0c/volumes" Jul 19 14:49:25.294509 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:49:25.289271 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:49:44.392170 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:49:44.392134 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:49:49.596320 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:49:49.596285 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:50:11.696375 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.696337 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk"] Jul 19 14:50:11.696860 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.696840 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2e5720-61ad-446b-9282-9d14f5415c0c" containerName="authorino" Jul 19 14:50:11.696931 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.696882 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2e5720-61ad-446b-9282-9d14f5415c0c" containerName="authorino" Jul 19 14:50:11.697021 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.697008 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2e5720-61ad-446b-9282-9d14f5415c0c" containerName="authorino" Jul 19 14:50:11.700323 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.700304 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.702782 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.702760 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 19 14:50:11.703755 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.703733 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 19 14:50:11.703915 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.703895 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kzk9c\"" Jul 19 14:50:11.703996 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.703953 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 19 14:50:11.709002 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.708978 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk"] Jul 19 14:50:11.868846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.868812 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.868846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.868849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf39c267-14d5-4bfd-ad66-a022327fd493-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.869056 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.868922 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.869056 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.868942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqsv\" (UniqueName: \"kubernetes.io/projected/cf39c267-14d5-4bfd-ad66-a022327fd493-kube-api-access-9qqsv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.869056 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.868994 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.869056 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.869011 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.970075 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.969985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.970075 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.970023 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf39c267-14d5-4bfd-ad66-a022327fd493-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.970271 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.970105 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.970271 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.970130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqsv\" (UniqueName: \"kubernetes.io/projected/cf39c267-14d5-4bfd-ad66-a022327fd493-kube-api-access-9qqsv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.970271 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.970215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.970271 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.970252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.970529 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.970509 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.970590 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.970531 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.970590 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.970580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.972396 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.972369 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf39c267-14d5-4bfd-ad66-a022327fd493-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.972736 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.972717 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf39c267-14d5-4bfd-ad66-a022327fd493-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:11.977563 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:11.977542 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqsv\" (UniqueName: \"kubernetes.io/projected/cf39c267-14d5-4bfd-ad66-a022327fd493-kube-api-access-9qqsv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6rjk\" (UID: \"cf39c267-14d5-4bfd-ad66-a022327fd493\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:12.012069 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:12.012034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:12.146768 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:12.146743 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk"] Jul 19 14:50:12.148496 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:50:12.148457 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf39c267_14d5_4bfd_ad66_a022327fd493.slice/crio-e237955f30f457d0468b31482efb61cb526a178b238edc5c4f363246a28e1289 WatchSource:0}: Error finding container e237955f30f457d0468b31482efb61cb526a178b238edc5c4f363246a28e1289: Status 404 returned error can't find the container with id e237955f30f457d0468b31482efb61cb526a178b238edc5c4f363246a28e1289 Jul 19 14:50:12.486319 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:12.486232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" event={"ID":"cf39c267-14d5-4bfd-ad66-a022327fd493","Type":"ContainerStarted","Data":"e237955f30f457d0468b31482efb61cb526a178b238edc5c4f363246a28e1289"} Jul 19 14:50:13.003581 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:13.003544 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:50:13.493091 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:13.493056 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" event={"ID":"cf39c267-14d5-4bfd-ad66-a022327fd493","Type":"ContainerStarted","Data":"5cddcd5719df05333d4f9afce4cbad00968b9ba1dcfc5f51ede0f65457c80ba2"} Jul 19 14:50:13.511776 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:13.511681 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" podStartSLOduration=1.408869782 podStartE2EDuration="2.511666096s" totalImagesPullingTime="1.102796314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:50:13.50973044 +0000 UTC m=+825.931954771" watchObservedRunningTime="2026-07-19 14:50:13.511666096 +0000 UTC m=+825.933890428" Jul 19 14:50:22.013155 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:22.013118 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:22.013575 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:22.013439 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:22.025956 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:22.025929 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:22.197598 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:22.197564 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:50:22.547105 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:22.547072 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6rjk" Jul 19 14:50:53.395485 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:53.395446 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:50:56.895639 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:56.895599 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk"] Jul 19 14:50:56.899096 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:56.899073 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:56.901673 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:56.901652 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 19 14:50:56.908265 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:56.908240 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk"] Jul 19 14:50:57.050730 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.050694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.050916 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.050744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zllr\" (UniqueName: \"kubernetes.io/projected/78f3c579-56f0-4e60-9d81-e1ddaab26c08-kube-api-access-9zllr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.050916 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.050817 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.050916 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.050882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.051028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.050916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.051028 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.050936 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f3c579-56f0-4e60-9d81-e1ddaab26c08-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.151460 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.151376 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.151460 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.151418 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zllr\" (UniqueName: \"kubernetes.io/projected/78f3c579-56f0-4e60-9d81-e1ddaab26c08-kube-api-access-9zllr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.151684 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.151540 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.151684 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.151588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.151684 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.151639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.151684 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.151670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f3c579-56f0-4e60-9d81-e1ddaab26c08-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.151921 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.151824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.151978 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.151958 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.152020 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.151966 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.153854 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.153835 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78f3c579-56f0-4e60-9d81-e1ddaab26c08-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.154141 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.154124 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f3c579-56f0-4e60-9d81-e1ddaab26c08-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.158956 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.158935 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zllr\" (UniqueName: \"kubernetes.io/projected/78f3c579-56f0-4e60-9d81-e1ddaab26c08-kube-api-access-9zllr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk\" (UID: \"78f3c579-56f0-4e60-9d81-e1ddaab26c08\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.207967 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.207943 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:50:57.337846 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.337817 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk"] Jul 19 14:50:57.339421 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:50:57.339392 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78f3c579_56f0_4e60_9d81_e1ddaab26c08.slice/crio-6c8e7f138b164cd4875ba1fc65cc70293240a54bf2b0a05a62cbe9400eb8f2fb WatchSource:0}: Error finding container 6c8e7f138b164cd4875ba1fc65cc70293240a54bf2b0a05a62cbe9400eb8f2fb: Status 404 returned error can't find the container with id 6c8e7f138b164cd4875ba1fc65cc70293240a54bf2b0a05a62cbe9400eb8f2fb Jul 19 14:50:57.707590 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.707550 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" event={"ID":"78f3c579-56f0-4e60-9d81-e1ddaab26c08","Type":"ContainerStarted","Data":"85e60145b63670a0008e310d66d963790a1af21ff83e15e2c565ae7e57619f63"} Jul 19 14:50:57.707590 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.707597 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" event={"ID":"78f3c579-56f0-4e60-9d81-e1ddaab26c08","Type":"ContainerStarted","Data":"6c8e7f138b164cd4875ba1fc65cc70293240a54bf2b0a05a62cbe9400eb8f2fb"} Jul 19 14:50:57.725596 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:50:57.725552 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" podStartSLOduration=1.549611114 podStartE2EDuration="1.725537719s" totalImagesPullingTime="175.926605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:50:57.724948056 +0000 UTC m=+870.147172386" watchObservedRunningTime="2026-07-19 14:50:57.725537719 +0000 UTC m=+870.147762094" Jul 19 14:51:07.208616 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:07.208575 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:51:07.208616 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:07.208624 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:51:07.220920 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:07.220897 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:51:07.777979 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:07.777936 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk" Jul 19 14:51:08.090608 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:08.090563 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:51:28.079376 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:28.079286 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:51:28.082563 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:28.082538 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:51:47.677326 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.677286 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77c6ff4774-ngpxf"] Jul 19 14:51:47.680834 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.680812 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77c6ff4774-ngpxf" Jul 19 14:51:47.684040 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.684023 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 19 14:51:47.684134 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.684024 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k9sdp\"" Jul 19 14:51:47.688453 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.688433 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77c6ff4774-ngpxf"] Jul 19 14:51:47.780590 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.780551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2dk7\" (UniqueName: \"kubernetes.io/projected/fb919240-ff76-4e7a-847c-48a8f7f48777-kube-api-access-l2dk7\") pod \"authorino-77c6ff4774-ngpxf\" (UID: \"fb919240-ff76-4e7a-847c-48a8f7f48777\") " pod="kuadrant-system/authorino-77c6ff4774-ngpxf" Jul 19 14:51:47.780768 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.780617 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb919240-ff76-4e7a-847c-48a8f7f48777-tls-cert\") pod \"authorino-77c6ff4774-ngpxf\" (UID: \"fb919240-ff76-4e7a-847c-48a8f7f48777\") " pod="kuadrant-system/authorino-77c6ff4774-ngpxf" Jul 19 14:51:47.882027 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.881987 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2dk7\" (UniqueName: \"kubernetes.io/projected/fb919240-ff76-4e7a-847c-48a8f7f48777-kube-api-access-l2dk7\") pod \"authorino-77c6ff4774-ngpxf\" (UID: \"fb919240-ff76-4e7a-847c-48a8f7f48777\") " pod="kuadrant-system/authorino-77c6ff4774-ngpxf" Jul 19 14:51:47.882233 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.882153 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb919240-ff76-4e7a-847c-48a8f7f48777-tls-cert\") pod \"authorino-77c6ff4774-ngpxf\" (UID: \"fb919240-ff76-4e7a-847c-48a8f7f48777\") " pod="kuadrant-system/authorino-77c6ff4774-ngpxf" Jul 19 14:51:47.884581 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.884557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb919240-ff76-4e7a-847c-48a8f7f48777-tls-cert\") pod \"authorino-77c6ff4774-ngpxf\" (UID: \"fb919240-ff76-4e7a-847c-48a8f7f48777\") " pod="kuadrant-system/authorino-77c6ff4774-ngpxf" Jul 19 14:51:47.889365 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.889342 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2dk7\" (UniqueName: \"kubernetes.io/projected/fb919240-ff76-4e7a-847c-48a8f7f48777-kube-api-access-l2dk7\") pod \"authorino-77c6ff4774-ngpxf\" (UID: \"fb919240-ff76-4e7a-847c-48a8f7f48777\") " pod="kuadrant-system/authorino-77c6ff4774-ngpxf" Jul 19 14:51:47.991313 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:47.991243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77c6ff4774-ngpxf" Jul 19 14:51:48.122913 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:48.122889 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77c6ff4774-ngpxf"] Jul 19 14:51:48.124557 ip-10-0-134-156 kubenswrapper[2559]: W0719 14:51:48.124534 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb919240_ff76_4e7a_847c_48a8f7f48777.slice/crio-6fef10b142d0a70c586df014e4f165cb98f75456e1d5f478ccde417c2eb3e209 WatchSource:0}: Error finding container 6fef10b142d0a70c586df014e4f165cb98f75456e1d5f478ccde417c2eb3e209: Status 404 returned error can't find the container with id 6fef10b142d0a70c586df014e4f165cb98f75456e1d5f478ccde417c2eb3e209 Jul 19 14:51:48.970263 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:48.970161 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77c6ff4774-ngpxf" event={"ID":"fb919240-ff76-4e7a-847c-48a8f7f48777","Type":"ContainerStarted","Data":"013efb25cfe16415ef1ad3e7d0027de8089f2ad961f89f482a8392a44ad18ec4"} Jul 19 14:51:48.970263 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:48.970214 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77c6ff4774-ngpxf" event={"ID":"fb919240-ff76-4e7a-847c-48a8f7f48777","Type":"ContainerStarted","Data":"6fef10b142d0a70c586df014e4f165cb98f75456e1d5f478ccde417c2eb3e209"} Jul 19 14:51:48.989789 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:51:48.989738 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77c6ff4774-ngpxf" podStartSLOduration=1.436632666 podStartE2EDuration="1.989724296s" totalImagesPullingTime="553.09163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:51:48.988459101 +0000 UTC m=+921.410683431" watchObservedRunningTime="2026-07-19 14:51:48.989724296 +0000 UTC m=+921.411948626" Jul 19 14:52:39.399912 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:52:39.399857 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:52:48.405704 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:52:48.405666 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:52:59.692606 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:52:59.692531 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:54:41.795723 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:54:41.795685 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:54:51.446548 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:54:51.446506 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:55:19.596371 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:55:19.596336 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:55:28.997745 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:55:28.997706 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:55:40.210964 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:55:40.210923 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:55:47.898658 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:55:47.898615 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:55:59.295932 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:55:59.295834 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:56:09.901334 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:56:09.901299 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:56:20.499132 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:56:20.499090 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:56:28.114738 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:56:28.114711 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:56:28.118460 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:56:28.118438 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 14:56:30.508047 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:56:30.508016 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:56:55.692021 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:56:55.691985 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:57:07.798300 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:57:07.798262 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:57:30.608418 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:57:30.608386 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:57:41.296857 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:57:41.296823 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:57:56.609716 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:57:56.609673 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:58:06.421275 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:58:06.416046 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:58:15.904160 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:58:15.904125 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:58:26.095398 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:58:26.095361 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 14:58:35.802711 ip-10-0-134-156 kubenswrapper[2559]: I0719 14:58:35.802674 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:00:23.198584 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:00:23.198484 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:00:32.902699 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:00:32.902664 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:00:45.902100 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:00:45.902051 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:00:59.101501 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:00:59.101465 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:01:10.295174 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:01:10.295139 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:01:21.451891 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:01:21.451845 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:01:28.147942 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:01:28.147910 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:01:28.160591 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:01:28.160564 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:01:29.391680 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:01:29.391645 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:01:40.801981 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:01:40.801943 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:01:51.793118 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:01:51.793033 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:02:00.597096 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:00.597059 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:02:11.490923 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:11.490886 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:02:33.393014 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:33.392979 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:02:40.984781 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:40.984748 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw"] Jul 19 15:02:40.988241 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:40.988217 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:40.990535 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:40.990508 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-89c2edb3-kserve-self-signed-certs\"" Jul 19 15:02:41.008231 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.008199 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw"] Jul 19 15:02:41.090082 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.090042 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tls-certs\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.090273 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.090107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tmp-dir\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.090273 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.090179 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-dshm\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.090273 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.090227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-home\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.090273 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.090271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-model-cache\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.090415 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.090318 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxjz\" (UniqueName: \"kubernetes.io/projected/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-kube-api-access-wfxjz\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.191038 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.190992 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-home\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.191218 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.191064 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-model-cache\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.191218 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.191099 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxjz\" (UniqueName: \"kubernetes.io/projected/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-kube-api-access-wfxjz\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.191218 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.191157 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tls-certs\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.191218 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.191197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tmp-dir\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.191425 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.191250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-dshm\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.191509 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.191487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-home\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.191571 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.191518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-model-cache\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.191623 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.191568 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tmp-dir\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.193537 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.193519 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-dshm\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.193617 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.193593 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tls-certs\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.198670 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.198650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxjz\" (UniqueName: \"kubernetes.io/projected/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-kube-api-access-wfxjz\") pod \"e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.299845 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.299745 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:41.437072 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:41.437044 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw"] Jul 19 15:02:41.438778 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:02:41.438751 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a8894a0_eb3f_4f8f_a65c_39ef5602853a.slice/crio-24182331e612312c921babc95bda509ef2dbe65c3d6016ffbccd98725b6d0c22 WatchSource:0}: Error finding container 24182331e612312c921babc95bda509ef2dbe65c3d6016ffbccd98725b6d0c22: Status 404 returned error can't find the container with id 24182331e612312c921babc95bda509ef2dbe65c3d6016ffbccd98725b6d0c22 Jul 19 15:02:42.334533 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:42.334494 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" event={"ID":"1a8894a0-eb3f-4f8f-a65c-39ef5602853a","Type":"ContainerStarted","Data":"f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda"} Jul 19 15:02:42.334988 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:42.334542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" event={"ID":"1a8894a0-eb3f-4f8f-a65c-39ef5602853a","Type":"ContainerStarted","Data":"24182331e612312c921babc95bda509ef2dbe65c3d6016ffbccd98725b6d0c22"} Jul 19 15:02:42.353268 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:42.353224 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" podStartSLOduration=2.353209233 podStartE2EDuration="2.353209233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:02:42.351446665 +0000 UTC m=+1574.773670998" watchObservedRunningTime="2026-07-19 15:02:42.353209233 +0000 UTC m=+1574.775433565" Jul 19 15:02:43.798294 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:43.798255 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:02:51.300252 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:51.300148 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:51.300252 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:51.300214 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:51.313137 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:51.313101 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:02:51.398693 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:02:51.398663 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:03:09.499322 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:09.499285 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:03:27.916855 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:27.916758 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw"] Jul 19 15:03:31.991885 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:31.991835 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:03:42.937831 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:42.937768 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" podUID="1a8894a0-eb3f-4f8f-a65c-39ef5602853a" containerName="main" containerID="cri-o://f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda" gracePeriod=45 Jul 19 15:03:43.181604 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.181580 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:03:43.346582 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.346550 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-home\") pod \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " Jul 19 15:03:43.346778 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.346632 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tls-certs\") pod \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " Jul 19 15:03:43.346778 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.346656 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfxjz\" (UniqueName: \"kubernetes.io/projected/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-kube-api-access-wfxjz\") pod \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " Jul 19 15:03:43.346778 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.346684 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-model-cache\") pod \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " Jul 19 15:03:43.346778 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.346722 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tmp-dir\") pod \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " Jul 19 15:03:43.347032 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.346789 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-dshm\") pod \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\" (UID: \"1a8894a0-eb3f-4f8f-a65c-39ef5602853a\") " Jul 19 15:03:43.347032 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.346916 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-home" (OuterVolumeSpecName: "home") pod "1a8894a0-eb3f-4f8f-a65c-39ef5602853a" (UID: "1a8894a0-eb3f-4f8f-a65c-39ef5602853a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:03:43.347145 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.347046 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a8894a0-eb3f-4f8f-a65c-39ef5602853a" (UID: "1a8894a0-eb3f-4f8f-a65c-39ef5602853a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:03:43.347145 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.347070 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a8894a0-eb3f-4f8f-a65c-39ef5602853a" (UID: "1a8894a0-eb3f-4f8f-a65c-39ef5602853a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:03:43.347145 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.347101 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:03:43.347145 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.347116 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:03:43.349029 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.348995 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-dshm" (OuterVolumeSpecName: "dshm") pod "1a8894a0-eb3f-4f8f-a65c-39ef5602853a" (UID: "1a8894a0-eb3f-4f8f-a65c-39ef5602853a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:03:43.349145 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.349100 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a8894a0-eb3f-4f8f-a65c-39ef5602853a" (UID: "1a8894a0-eb3f-4f8f-a65c-39ef5602853a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:03:43.349207 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.349151 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-kube-api-access-wfxjz" (OuterVolumeSpecName: "kube-api-access-wfxjz") pod "1a8894a0-eb3f-4f8f-a65c-39ef5602853a" (UID: "1a8894a0-eb3f-4f8f-a65c-39ef5602853a"). InnerVolumeSpecName "kube-api-access-wfxjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:03:43.448285 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.448234 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:03:43.448285 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.448281 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfxjz\" (UniqueName: \"kubernetes.io/projected/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-kube-api-access-wfxjz\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:03:43.448285 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.448292 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:03:43.448285 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.448304 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8894a0-eb3f-4f8f-a65c-39ef5602853a-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:03:43.657335 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.657245 2559 generic.go:356] "Generic (PLEG): container finished" podID="1a8894a0-eb3f-4f8f-a65c-39ef5602853a" containerID="f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda" exitCode=0 Jul 19 15:03:43.657335 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.657286 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" Jul 19 15:03:43.657543 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.657325 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" event={"ID":"1a8894a0-eb3f-4f8f-a65c-39ef5602853a","Type":"ContainerDied","Data":"f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda"} Jul 19 15:03:43.657543 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.657369 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw" event={"ID":"1a8894a0-eb3f-4f8f-a65c-39ef5602853a","Type":"ContainerDied","Data":"24182331e612312c921babc95bda509ef2dbe65c3d6016ffbccd98725b6d0c22"} Jul 19 15:03:43.657543 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.657387 2559 scope.go:117] "RemoveContainer" containerID="f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda" Jul 19 15:03:43.669748 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.669730 2559 scope.go:117] "RemoveContainer" containerID="f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda" Jul 19 15:03:43.670051 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:03:43.670031 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda\": container with ID starting with f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda not found: ID does not exist" containerID="f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda" Jul 19 15:03:43.670145 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.670063 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda"} err="failed to get container status \"f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda\": rpc error: code = NotFound desc = could not find container \"f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda\": container with ID starting with f41f42a21795efa8fa1f2eaa09da5ef5505156901f5236c027751fab54b10bda not found: ID does not exist" Jul 19 15:03:43.679173 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.679150 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw"] Jul 19 15:03:43.682990 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:43.682968 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-89c2edb3-kserve-67c89bfb55-8pdcw"] Jul 19 15:03:44.106669 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:44.106635 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8894a0-eb3f-4f8f-a65c-39ef5602853a" path="/var/lib/kubelet/pods/1a8894a0-eb3f-4f8f-a65c-39ef5602853a/volumes" Jul 19 15:03:45.296940 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:45.296903 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:03:54.597004 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:03:54.596969 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:04:07.197878 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:04:07.197833 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:04:17.798280 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:04:17.798241 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:04:27.394906 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:04:27.394856 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:04:36.591823 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:04:36.591783 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:04:47.606066 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:04:47.606023 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:04:58.607907 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:04:58.607799 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:05:10.595811 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:05:10.595762 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:05:25.199671 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:05:25.199637 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:05:34.802700 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:05:34.802665 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:05:41.698227 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:05:41.698188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:05:53.801447 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:05:53.801404 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:06:02.100292 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:06:02.100255 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:06:14.898196 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:06:14.898161 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:06:26.000452 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:06:26.000368 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:06:28.189034 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:06:28.188994 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:06:28.195449 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:06:28.195421 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:06:37.500485 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:06:37.500443 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:07:00.301581 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:07:00.301542 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:07:31.692252 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:07:31.692210 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:07:43.995014 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:07:43.994960 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:08:15.394390 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:08:15.394343 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:08:26.994303 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:08:26.994264 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:08:59.500989 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:08:59.500949 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:09:11.300673 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:09:11.300633 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:09:44.094192 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:09:44.094150 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:11:11.193310 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:11:11.193221 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:11:17.594458 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:11:17.594423 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:11:26.397404 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:11:26.397360 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:11:28.223038 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:11:28.223009 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:11:28.231315 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:11:28.231290 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:11:47.797546 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:11:47.797508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:12:26.698208 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:26.698112 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:12:33.694014 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.693978 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns"] Jul 19 15:12:33.696212 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.694312 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8894a0-eb3f-4f8f-a65c-39ef5602853a" containerName="main" Jul 19 15:12:33.696212 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.694323 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8894a0-eb3f-4f8f-a65c-39ef5602853a" containerName="main" Jul 19 15:12:33.696212 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.694391 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8894a0-eb3f-4f8f-a65c-39ef5602853a" containerName="main" Jul 19 15:12:33.697062 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.697045 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.700384 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.700360 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c515b9b9\"/\"openshift-service-ca.crt\"" Jul 19 15:12:33.700497 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.700407 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c515b9b9\"/\"kube-root-ca.crt\"" Jul 19 15:12:33.700497 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.700371 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c515b9b9\"/\"default-dockercfg-nzgpm\"" Jul 19 15:12:33.700497 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.700464 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c515b9b9\"/\"e2e-lifecycle-model-c515b9b9-kserve-self-signed-certs\"" Jul 19 15:12:33.704949 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.704902 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns"] Jul 19 15:12:33.843304 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.843257 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-home\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.843507 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.843366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tmp-dir\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.843507 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.843404 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-dshm\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.843507 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.843445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tls-certs\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.843647 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.843554 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d726g\" (UniqueName: \"kubernetes.io/projected/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-kube-api-access-d726g\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.843647 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.843585 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-model-cache\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.944761 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.944682 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-home\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.944761 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.944722 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tmp-dir\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.944761 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.944741 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-dshm\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.945047 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.944768 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tls-certs\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.945047 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.944804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d726g\" (UniqueName: \"kubernetes.io/projected/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-kube-api-access-d726g\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.945047 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.944826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-model-cache\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.945204 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.945169 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-home\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.945265 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.945204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-model-cache\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.945327 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.945295 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tmp-dir\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.947148 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.947127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-dshm\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.947291 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.947276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tls-certs\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:33.952439 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:33.952416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d726g\" (UniqueName: \"kubernetes.io/projected/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-kube-api-access-d726g\") pod \"e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:34.007232 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:34.007205 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:34.132776 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:34.132754 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns"] Jul 19 15:12:34.134591 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:12:34.134568 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b477799_17f7_4784_b5b9_ac5dd7ce92f0.slice/crio-8e914f8244b84c925d8696fb3d8df1a3d932dc21e8d1a749d8700b327f0119c4 WatchSource:0}: Error finding container 8e914f8244b84c925d8696fb3d8df1a3d932dc21e8d1a749d8700b327f0119c4: Status 404 returned error can't find the container with id 8e914f8244b84c925d8696fb3d8df1a3d932dc21e8d1a749d8700b327f0119c4 Jul 19 15:12:34.411627 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:34.411584 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" event={"ID":"4b477799-17f7-4784-b5b9-ac5dd7ce92f0","Type":"ContainerStarted","Data":"947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8"} Jul 19 15:12:34.411797 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:34.411639 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" event={"ID":"4b477799-17f7-4784-b5b9-ac5dd7ce92f0","Type":"ContainerStarted","Data":"8e914f8244b84c925d8696fb3d8df1a3d932dc21e8d1a749d8700b327f0119c4"} Jul 19 15:12:34.428567 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:34.428517 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" podStartSLOduration=1.428499227 podStartE2EDuration="1.428499227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:12:34.426277151 +0000 UTC m=+2166.848501493" watchObservedRunningTime="2026-07-19 15:12:34.428499227 +0000 UTC m=+2166.850723559" Jul 19 15:12:39.208495 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:39.208459 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:12:44.008110 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:44.008071 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:44.008504 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:44.008126 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:44.020672 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:44.020644 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:44.474975 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:44.474949 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:12:48.600368 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:12:48.600323 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:13:07.097655 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:07.097617 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:13:15.994720 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:15.994685 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:13:20.107601 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:20.107567 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns"] Jul 19 15:13:35.128812 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.128760 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" podUID="4b477799-17f7-4784-b5b9-ac5dd7ce92f0" containerName="main" containerID="cri-o://947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8" gracePeriod=45 Jul 19 15:13:35.380600 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.380530 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:13:35.481598 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.481559 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tmp-dir\") pod \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " Jul 19 15:13:35.481784 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.481620 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-home\") pod \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " Jul 19 15:13:35.481784 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.481645 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-model-cache\") pod \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " Jul 19 15:13:35.481784 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.481676 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-dshm\") pod \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " Jul 19 15:13:35.481784 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.481765 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tls-certs\") pod \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " Jul 19 15:13:35.482036 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.481827 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d726g\" (UniqueName: \"kubernetes.io/projected/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-kube-api-access-d726g\") pod \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\" (UID: \"4b477799-17f7-4784-b5b9-ac5dd7ce92f0\") " Jul 19 15:13:35.482036 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.481885 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-home" (OuterVolumeSpecName: "home") pod "4b477799-17f7-4784-b5b9-ac5dd7ce92f0" (UID: "4b477799-17f7-4784-b5b9-ac5dd7ce92f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:13:35.482036 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.481897 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b477799-17f7-4784-b5b9-ac5dd7ce92f0" (UID: "4b477799-17f7-4784-b5b9-ac5dd7ce92f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:13:35.482036 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.481920 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b477799-17f7-4784-b5b9-ac5dd7ce92f0" (UID: "4b477799-17f7-4784-b5b9-ac5dd7ce92f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:13:35.482241 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.482154 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:13:35.482241 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.482178 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:13:35.482241 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.482194 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:13:35.484030 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.484001 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b477799-17f7-4784-b5b9-ac5dd7ce92f0" (UID: "4b477799-17f7-4784-b5b9-ac5dd7ce92f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:13:35.484030 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.484000 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-kube-api-access-d726g" (OuterVolumeSpecName: "kube-api-access-d726g") pod "4b477799-17f7-4784-b5b9-ac5dd7ce92f0" (UID: "4b477799-17f7-4784-b5b9-ac5dd7ce92f0"). InnerVolumeSpecName "kube-api-access-d726g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:13:35.484177 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.484109 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-dshm" (OuterVolumeSpecName: "dshm") pod "4b477799-17f7-4784-b5b9-ac5dd7ce92f0" (UID: "4b477799-17f7-4784-b5b9-ac5dd7ce92f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:13:35.582861 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.582817 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:13:35.582861 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.582850 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:13:35.582861 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.582861 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d726g\" (UniqueName: \"kubernetes.io/projected/4b477799-17f7-4784-b5b9-ac5dd7ce92f0-kube-api-access-d726g\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:13:35.731106 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.731006 2559 generic.go:356] "Generic (PLEG): container finished" podID="4b477799-17f7-4784-b5b9-ac5dd7ce92f0" containerID="947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8" exitCode=0 Jul 19 15:13:35.731106 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.731051 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" Jul 19 15:13:35.731106 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.731087 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" event={"ID":"4b477799-17f7-4784-b5b9-ac5dd7ce92f0","Type":"ContainerDied","Data":"947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8"} Jul 19 15:13:35.731349 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.731130 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns" event={"ID":"4b477799-17f7-4784-b5b9-ac5dd7ce92f0","Type":"ContainerDied","Data":"8e914f8244b84c925d8696fb3d8df1a3d932dc21e8d1a749d8700b327f0119c4"} Jul 19 15:13:35.731349 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.731147 2559 scope.go:117] "RemoveContainer" containerID="947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8" Jul 19 15:13:35.744832 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.744813 2559 scope.go:117] "RemoveContainer" containerID="947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8" Jul 19 15:13:35.745103 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:13:35.745083 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8\": container with ID starting with 947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8 not found: ID does not exist" containerID="947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8" Jul 19 15:13:35.745176 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.745111 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8"} err="failed to get container status \"947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8\": rpc error: code = NotFound desc = could not find container \"947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8\": container with ID starting with 947df1a5237bb92cdab05f78812628e12b75ccf217819e1262ebc522ebaebfa8 not found: ID does not exist" Jul 19 15:13:35.754498 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.754475 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns"] Jul 19 15:13:35.756555 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:35.756532 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c515b9b9/e2e-lifecycle-model-c515b9b9-kserve-7cbb7965ff-xk5ns"] Jul 19 15:13:36.111057 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:13:36.108536 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b477799-17f7-4784-b5b9-ac5dd7ce92f0" path="/var/lib/kubelet/pods/4b477799-17f7-4784-b5b9-ac5dd7ce92f0/volumes" Jul 19 15:14:06.301163 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:14:06.301084 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:14:18.720280 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:14:18.720240 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:15:08.611181 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:08.611141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:15:23.193311 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:23.193224 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:15:32.789413 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.789375 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v"] Jul 19 15:15:32.789800 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.789714 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b477799-17f7-4784-b5b9-ac5dd7ce92f0" containerName="main" Jul 19 15:15:32.789800 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.789726 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b477799-17f7-4784-b5b9-ac5dd7ce92f0" containerName="main" Jul 19 15:15:32.789800 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.789788 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b477799-17f7-4784-b5b9-ac5dd7ce92f0" containerName="main" Jul 19 15:15:32.792948 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.792930 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:32.796325 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.796302 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f72db2\"/\"test-model-f72db2-kserve-self-signed-certs\"" Jul 19 15:15:32.796508 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.796486 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f72db2\"/\"default-dockercfg-rvfgh\"" Jul 19 15:15:32.796704 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.796429 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f72db2\"/\"kube-root-ca.crt\"" Jul 19 15:15:32.796800 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.796345 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f72db2\"/\"openshift-service-ca.crt\"" Jul 19 15:15:32.799108 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.799084 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v"] Jul 19 15:15:32.910692 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.910661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tmp-dir\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:32.910850 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.910729 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-model-cache\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:32.910850 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.910748 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glslh\" (UniqueName: \"kubernetes.io/projected/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-kube-api-access-glslh\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:32.910850 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.910767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-dshm\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:32.910850 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.910788 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-home\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:32.911032 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:32.910911 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tls-certs\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.011507 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.011463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-model-cache\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.011507 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.011508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glslh\" (UniqueName: \"kubernetes.io/projected/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-kube-api-access-glslh\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.011764 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.011533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-dshm\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.011764 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.011555 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-home\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.011764 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.011592 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tls-certs\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.011764 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.011633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tmp-dir\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.012108 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.012072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-home\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.012108 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.012090 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tmp-dir\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.012268 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.012150 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-model-cache\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.013849 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.013828 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-dshm\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.014035 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.014017 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tls-certs\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.018859 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.018837 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glslh\" (UniqueName: \"kubernetes.io/projected/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-kube-api-access-glslh\") pod \"test-model-f72db2-kserve-5cbf96dbdc-q985v\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.102609 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.102584 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:33.230657 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.230628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v"] Jul 19 15:15:33.232330 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:15:33.232292 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9a8461e_78c8_4a06_b547_5cabb6d1eeb6.slice/crio-5c4375e195a7c96a97c1eb2836c694c29296e215b8707406f685c822a234bbe1 WatchSource:0}: Error finding container 5c4375e195a7c96a97c1eb2836c694c29296e215b8707406f685c822a234bbe1: Status 404 returned error can't find the container with id 5c4375e195a7c96a97c1eb2836c694c29296e215b8707406f685c822a234bbe1 Jul 19 15:15:33.340600 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.340567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" event={"ID":"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6","Type":"ContainerStarted","Data":"9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58"} Jul 19 15:15:33.340751 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.340608 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" event={"ID":"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6","Type":"ContainerStarted","Data":"5c4375e195a7c96a97c1eb2836c694c29296e215b8707406f685c822a234bbe1"} Jul 19 15:15:33.357557 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:33.357473 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" podStartSLOduration=1.357459574 podStartE2EDuration="1.357459574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:15:33.355540714 +0000 UTC m=+2345.777765048" watchObservedRunningTime="2026-07-19 15:15:33.357459574 +0000 UTC m=+2345.779683884" Jul 19 15:15:37.401308 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:37.401266 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:15:43.103486 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:43.103447 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:43.104021 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:43.103500 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:43.115979 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:43.115947 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:43.402529 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:43.402455 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:15:54.997296 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:54.997257 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5"] Jul 19 15:15:55.000852 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.000830 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.003325 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.003299 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-36dc1d\"/\"kube-root-ca.crt\"" Jul 19 15:15:55.003449 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.003357 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-36dc1d\"/\"openshift-service-ca.crt\"" Jul 19 15:15:55.004189 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.004166 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-36dc1d\"/\"test-model-36dc1d-kserve-self-signed-certs\"" Jul 19 15:15:55.004302 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.004225 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-36dc1d\"/\"default-dockercfg-rq4t6\"" Jul 19 15:15:55.007602 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.007558 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5"] Jul 19 15:15:55.019674 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.019653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-model-cache\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.019765 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.019693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-dshm\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.019765 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.019719 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-tmp-dir\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.019843 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.019786 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c30a752-77c7-444b-a15b-5d9c54d41495-tls-certs\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.019843 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.019823 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr682\" (UniqueName: \"kubernetes.io/projected/6c30a752-77c7-444b-a15b-5d9c54d41495-kube-api-access-qr682\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.019947 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.019924 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-home\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.121265 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.121230 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-home\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.121462 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.121286 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-model-cache\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.121462 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.121318 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-dshm\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.121462 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.121338 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-tmp-dir\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.121462 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.121362 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c30a752-77c7-444b-a15b-5d9c54d41495-tls-certs\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.121462 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.121382 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr682\" (UniqueName: \"kubernetes.io/projected/6c30a752-77c7-444b-a15b-5d9c54d41495-kube-api-access-qr682\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.121734 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.121707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-home\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.121835 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.121814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-model-cache\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.121911 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.121824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-tmp-dir\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.123603 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.123584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-dshm\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.123762 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.123745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c30a752-77c7-444b-a15b-5d9c54d41495-tls-certs\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.129101 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.129077 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr682\" (UniqueName: \"kubernetes.io/projected/6c30a752-77c7-444b-a15b-5d9c54d41495-kube-api-access-qr682\") pod \"test-model-36dc1d-kserve-59fd45d4f9-q6tn5\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.312477 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.312383 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:15:55.445808 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.445778 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5"] Jul 19 15:15:55.448198 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:15:55.448173 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c30a752_77c7_444b_a15b_5d9c54d41495.slice/crio-8e583ed0ca06c4788b3d0aa4b4492679c79ee15be54dcd1aeeb01a6fc39047c6 WatchSource:0}: Error finding container 8e583ed0ca06c4788b3d0aa4b4492679c79ee15be54dcd1aeeb01a6fc39047c6: Status 404 returned error can't find the container with id 8e583ed0ca06c4788b3d0aa4b4492679c79ee15be54dcd1aeeb01a6fc39047c6 Jul 19 15:15:55.455499 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:55.455391 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" event={"ID":"6c30a752-77c7-444b-a15b-5d9c54d41495","Type":"ContainerStarted","Data":"8e583ed0ca06c4788b3d0aa4b4492679c79ee15be54dcd1aeeb01a6fc39047c6"} Jul 19 15:15:56.462417 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:56.462373 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" event={"ID":"6c30a752-77c7-444b-a15b-5d9c54d41495","Type":"ContainerStarted","Data":"2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8"} Jul 19 15:15:56.479310 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:15:56.479262 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" podStartSLOduration=2.479245547 podStartE2EDuration="2.479245547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:15:56.478065041 +0000 UTC m=+2368.900289374" watchObservedRunningTime="2026-07-19 15:15:56.479245547 +0000 UTC m=+2368.901469880" Jul 19 15:16:00.501282 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:00.501249 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:16:05.312573 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:05.312542 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:16:05.313038 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:05.312613 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:16:05.325337 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:05.325315 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:16:05.520067 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:05.520034 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:16:26.890431 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:26.890396 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:16:28.258365 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:28.258335 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:16:28.266773 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:28.266752 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:16:32.783150 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:32.783110 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v"] Jul 19 15:16:34.001020 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:34.000990 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5"] Jul 19 15:16:39.496826 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:39.496789 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:16:47.805275 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:47.805235 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" podUID="c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" containerName="main" containerID="cri-o://9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58" gracePeriod=45 Jul 19 15:16:48.046775 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.046748 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:16:48.104652 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.104614 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-home\") pod \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " Jul 19 15:16:48.104817 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.104662 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tls-certs\") pod \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " Jul 19 15:16:48.104817 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.104699 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-dshm\") pod \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " Jul 19 15:16:48.104817 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.104755 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glslh\" (UniqueName: \"kubernetes.io/projected/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-kube-api-access-glslh\") pod \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " Jul 19 15:16:48.104817 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.104783 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-model-cache\") pod \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " Jul 19 15:16:48.104817 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.104809 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tmp-dir\") pod \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\" (UID: \"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6\") " Jul 19 15:16:48.105119 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.104937 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-home" (OuterVolumeSpecName: "home") pod "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" (UID: "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:16:48.105119 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.105103 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:48.105194 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.105109 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" (UID: "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:16:48.105194 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.105117 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" (UID: "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:16:48.106992 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.106959 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" (UID: "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:16:48.107111 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.107073 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-kube-api-access-glslh" (OuterVolumeSpecName: "kube-api-access-glslh") pod "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" (UID: "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6"). InnerVolumeSpecName "kube-api-access-glslh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:16:48.107176 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.107131 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-dshm" (OuterVolumeSpecName: "dshm") pod "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" (UID: "c9a8461e-78c8-4a06-b547-5cabb6d1eeb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:16:48.206646 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.206611 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glslh\" (UniqueName: \"kubernetes.io/projected/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-kube-api-access-glslh\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:48.206646 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.206643 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:48.206816 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.206655 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:48.206816 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.206667 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:48.206816 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.206678 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:48.739956 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.739909 2559 generic.go:356] "Generic (PLEG): container finished" podID="c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" containerID="9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58" exitCode=0 Jul 19 15:16:48.740139 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.739956 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" Jul 19 15:16:48.740139 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.739967 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" event={"ID":"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6","Type":"ContainerDied","Data":"9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58"} Jul 19 15:16:48.740139 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.740009 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v" event={"ID":"c9a8461e-78c8-4a06-b547-5cabb6d1eeb6","Type":"ContainerDied","Data":"5c4375e195a7c96a97c1eb2836c694c29296e215b8707406f685c822a234bbe1"} Jul 19 15:16:48.740139 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.740032 2559 scope.go:117] "RemoveContainer" containerID="9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58" Jul 19 15:16:48.748845 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.748825 2559 scope.go:117] "RemoveContainer" containerID="9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58" Jul 19 15:16:48.749169 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:16:48.749148 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58\": container with ID starting with 9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58 not found: ID does not exist" containerID="9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58" Jul 19 15:16:48.749221 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.749176 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58"} err="failed to get container status \"9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58\": rpc error: code = NotFound desc = could not find container \"9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58\": container with ID starting with 9de22d579241e9091bab2b259463cd4412f55031d6afb15bfceeae8ab7a05d58 not found: ID does not exist" Jul 19 15:16:48.756006 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.755985 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v"] Jul 19 15:16:48.759731 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:48.759712 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f72db2/test-model-f72db2-kserve-5cbf96dbdc-q985v"] Jul 19 15:16:49.021788 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.021695 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" podUID="6c30a752-77c7-444b-a15b-5d9c54d41495" containerName="main" containerID="cri-o://2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8" gracePeriod=45 Jul 19 15:16:49.198893 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.198851 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:16:49.280280 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.280208 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:16:49.316436 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.316405 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-model-cache\") pod \"6c30a752-77c7-444b-a15b-5d9c54d41495\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " Jul 19 15:16:49.316593 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.316445 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr682\" (UniqueName: \"kubernetes.io/projected/6c30a752-77c7-444b-a15b-5d9c54d41495-kube-api-access-qr682\") pod \"6c30a752-77c7-444b-a15b-5d9c54d41495\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " Jul 19 15:16:49.316593 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.316475 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-tmp-dir\") pod \"6c30a752-77c7-444b-a15b-5d9c54d41495\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " Jul 19 15:16:49.316722 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.316616 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-dshm\") pod \"6c30a752-77c7-444b-a15b-5d9c54d41495\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " Jul 19 15:16:49.316722 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.316670 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c30a752-77c7-444b-a15b-5d9c54d41495-tls-certs\") pod \"6c30a752-77c7-444b-a15b-5d9c54d41495\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " Jul 19 15:16:49.316722 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.316667 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c30a752-77c7-444b-a15b-5d9c54d41495" (UID: "6c30a752-77c7-444b-a15b-5d9c54d41495"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:16:49.316910 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.316734 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c30a752-77c7-444b-a15b-5d9c54d41495" (UID: "6c30a752-77c7-444b-a15b-5d9c54d41495"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:16:49.316910 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.316756 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-home\") pod \"6c30a752-77c7-444b-a15b-5d9c54d41495\" (UID: \"6c30a752-77c7-444b-a15b-5d9c54d41495\") " Jul 19 15:16:49.317144 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.317119 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:49.317240 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.317148 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:49.317418 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.317394 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-home" (OuterVolumeSpecName: "home") pod "6c30a752-77c7-444b-a15b-5d9c54d41495" (UID: "6c30a752-77c7-444b-a15b-5d9c54d41495"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:16:49.318961 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.318934 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-dshm" (OuterVolumeSpecName: "dshm") pod "6c30a752-77c7-444b-a15b-5d9c54d41495" (UID: "6c30a752-77c7-444b-a15b-5d9c54d41495"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:16:49.319064 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.318975 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c30a752-77c7-444b-a15b-5d9c54d41495-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c30a752-77c7-444b-a15b-5d9c54d41495" (UID: "6c30a752-77c7-444b-a15b-5d9c54d41495"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:16:49.319116 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.319065 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c30a752-77c7-444b-a15b-5d9c54d41495-kube-api-access-qr682" (OuterVolumeSpecName: "kube-api-access-qr682") pod "6c30a752-77c7-444b-a15b-5d9c54d41495" (UID: "6c30a752-77c7-444b-a15b-5d9c54d41495"). InnerVolumeSpecName "kube-api-access-qr682". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:16:49.417913 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.417856 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr682\" (UniqueName: \"kubernetes.io/projected/6c30a752-77c7-444b-a15b-5d9c54d41495-kube-api-access-qr682\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:49.417913 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.417908 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:49.418128 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.417922 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c30a752-77c7-444b-a15b-5d9c54d41495-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:49.418128 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.417934 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c30a752-77c7-444b-a15b-5d9c54d41495-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:16:49.748198 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.748159 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c30a752-77c7-444b-a15b-5d9c54d41495" containerID="2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8" exitCode=0 Jul 19 15:16:49.748395 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.748204 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" Jul 19 15:16:49.748395 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.748245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" event={"ID":"6c30a752-77c7-444b-a15b-5d9c54d41495","Type":"ContainerDied","Data":"2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8"} Jul 19 15:16:49.748395 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.748292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5" event={"ID":"6c30a752-77c7-444b-a15b-5d9c54d41495","Type":"ContainerDied","Data":"8e583ed0ca06c4788b3d0aa4b4492679c79ee15be54dcd1aeeb01a6fc39047c6"} Jul 19 15:16:49.748395 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.748316 2559 scope.go:117] "RemoveContainer" containerID="2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8" Jul 19 15:16:49.758456 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.758434 2559 scope.go:117] "RemoveContainer" containerID="2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8" Jul 19 15:16:49.758716 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:16:49.758697 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8\": container with ID starting with 2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8 not found: ID does not exist" containerID="2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8" Jul 19 15:16:49.758804 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.758723 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8"} err="failed to get container status \"2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8\": rpc error: code = NotFound desc = could not find container \"2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8\": container with ID starting with 2f6626c78fa28ea7e3ef39b6a956fbb167bb2984d9d8f28a8268d47060c16ac8 not found: ID does not exist" Jul 19 15:16:49.770338 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.770314 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5"] Jul 19 15:16:49.773718 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:49.773695 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-36dc1d/test-model-36dc1d-kserve-59fd45d4f9-q6tn5"] Jul 19 15:16:50.107642 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:50.107610 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c30a752-77c7-444b-a15b-5d9c54d41495" path="/var/lib/kubelet/pods/6c30a752-77c7-444b-a15b-5d9c54d41495/volumes" Jul 19 15:16:50.108046 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:16:50.108031 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" path="/var/lib/kubelet/pods/c9a8461e-78c8-4a06-b547-5cabb6d1eeb6/volumes" Jul 19 15:17:09.597636 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:17:09.597554 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:17:29.495508 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:17:29.495467 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:17:45.589929 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:17:45.589890 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:18:01.903621 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:18:01.903579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:18:30.291241 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:18:30.291206 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:18:45.401519 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:18:45.401476 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:19:00.902515 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:00.902475 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:19:10.897657 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.897624 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5"] Jul 19 15:19:10.898262 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.897976 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" containerName="main" Jul 19 15:19:10.898262 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.897988 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" containerName="main" Jul 19 15:19:10.898262 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.898003 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c30a752-77c7-444b-a15b-5d9c54d41495" containerName="main" Jul 19 15:19:10.898262 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.898008 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c30a752-77c7-444b-a15b-5d9c54d41495" containerName="main" Jul 19 15:19:10.898262 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.898063 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c30a752-77c7-444b-a15b-5d9c54d41495" containerName="main" Jul 19 15:19:10.898262 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.898073 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a8461e-78c8-4a06-b547-5cabb6d1eeb6" containerName="main" Jul 19 15:19:10.901177 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.901156 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:10.906380 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.906347 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7679e0\"/\"auth-test-model-7679e0-kserve-self-signed-certs\"" Jul 19 15:19:10.906380 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.906362 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7679e0\"/\"default-dockercfg-dcp4j\"" Jul 19 15:19:10.906552 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.906407 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7679e0\"/\"kube-root-ca.crt\"" Jul 19 15:19:10.906552 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.906366 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7679e0\"/\"openshift-service-ca.crt\"" Jul 19 15:19:10.911491 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:10.911473 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5"] Jul 19 15:19:11.027063 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.027025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-model-cache\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.027063 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.027065 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ml66\" (UniqueName: \"kubernetes.io/projected/cc2c0635-d4e6-41e7-93e0-31d361287ff2-kube-api-access-2ml66\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.027241 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.027147 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-home\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.027241 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.027180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tmp-dir\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.027314 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.027256 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-dshm\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.027314 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.027280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tls-certs\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.127829 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.127793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tls-certs\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.127829 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.127837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-model-cache\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.128061 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.127855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml66\" (UniqueName: \"kubernetes.io/projected/cc2c0635-d4e6-41e7-93e0-31d361287ff2-kube-api-access-2ml66\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.128061 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.127919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-home\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.128061 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.127936 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tmp-dir\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.128061 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.128000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-dshm\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.128293 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.128268 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-model-cache\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.128348 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.128336 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-home\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.128412 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.128383 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tmp-dir\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.130167 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.130150 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-dshm\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.130371 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.130351 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tls-certs\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.135825 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.135795 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ml66\" (UniqueName: \"kubernetes.io/projected/cc2c0635-d4e6-41e7-93e0-31d361287ff2-kube-api-access-2ml66\") pod \"auth-test-model-7679e0-kserve-65ddb56695-6lkz5\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.210747 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.210673 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:11.338683 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.338658 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5"] Jul 19 15:19:11.340347 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:19:11.340319 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc2c0635_d4e6_41e7_93e0_31d361287ff2.slice/crio-106c0cf92c5af0f9dc1f8f25560b3b17df54513ddffe6518361e5b6710f6cbf1 WatchSource:0}: Error finding container 106c0cf92c5af0f9dc1f8f25560b3b17df54513ddffe6518361e5b6710f6cbf1: Status 404 returned error can't find the container with id 106c0cf92c5af0f9dc1f8f25560b3b17df54513ddffe6518361e5b6710f6cbf1 Jul 19 15:19:11.481132 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.481046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" event={"ID":"cc2c0635-d4e6-41e7-93e0-31d361287ff2","Type":"ContainerStarted","Data":"b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd"} Jul 19 15:19:11.481132 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.481090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" event={"ID":"cc2c0635-d4e6-41e7-93e0-31d361287ff2","Type":"ContainerStarted","Data":"106c0cf92c5af0f9dc1f8f25560b3b17df54513ddffe6518361e5b6710f6cbf1"} Jul 19 15:19:11.497336 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:11.497291 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" podStartSLOduration=1.4972763150000001 podStartE2EDuration="1.497276315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:19:11.496472485 +0000 UTC m=+2563.918696809" watchObservedRunningTime="2026-07-19 15:19:11.497276315 +0000 UTC m=+2563.919500624" Jul 19 15:19:14.196458 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:14.196426 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:19:21.211305 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:21.211255 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:21.211684 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:21.211323 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:21.223892 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:21.223853 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:19:21.546051 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:19:21.545964 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:20:00.311003 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.310914 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj"] Jul 19 15:20:00.314399 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.314376 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.317141 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.317117 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"openshift-service-ca.crt\"" Jul 19 15:20:00.317254 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.317119 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"auth-test-model-3c4e71-kserve-self-signed-certs\"" Jul 19 15:20:00.318016 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.317991 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"kube-root-ca.crt\"" Jul 19 15:20:00.318129 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.318065 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"default-dockercfg-5zs5j\"" Jul 19 15:20:00.323953 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.323929 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj"] Jul 19 15:20:00.455193 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.455145 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-dshm\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.455403 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.455202 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-home\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.455403 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.455297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-tmp-dir\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.455403 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.455371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99cb155-0164-4395-b665-39cae0e4386e-tls-certs\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.455616 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.455500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scmgn\" (UniqueName: \"kubernetes.io/projected/c99cb155-0164-4395-b665-39cae0e4386e-kube-api-access-scmgn\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.455616 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.455601 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-model-cache\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.556242 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.556206 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-home\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.556242 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.556242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-tmp-dir\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.556558 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.556272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99cb155-0164-4395-b665-39cae0e4386e-tls-certs\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.556558 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.556342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scmgn\" (UniqueName: \"kubernetes.io/projected/c99cb155-0164-4395-b665-39cae0e4386e-kube-api-access-scmgn\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.556558 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.556397 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-model-cache\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.556558 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.556464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-dshm\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.556759 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.556681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-tmp-dir\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.556759 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.556714 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-home\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.556759 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.556739 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-model-cache\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.558660 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.558639 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-dshm\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.558927 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.558909 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99cb155-0164-4395-b665-39cae0e4386e-tls-certs\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.564603 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.564548 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scmgn\" (UniqueName: \"kubernetes.io/projected/c99cb155-0164-4395-b665-39cae0e4386e-kube-api-access-scmgn\") pod \"auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.625713 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.625675 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:00.762599 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:00.762571 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj"] Jul 19 15:20:00.764436 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:20:00.764408 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc99cb155_0164_4395_b665_39cae0e4386e.slice/crio-bdba8ae425dbe3c70ff0829e0cd69ac8f00cb9225e9c44248f52e15f9079f3ac WatchSource:0}: Error finding container bdba8ae425dbe3c70ff0829e0cd69ac8f00cb9225e9c44248f52e15f9079f3ac: Status 404 returned error can't find the container with id bdba8ae425dbe3c70ff0829e0cd69ac8f00cb9225e9c44248f52e15f9079f3ac Jul 19 15:20:01.742501 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:01.742461 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" event={"ID":"c99cb155-0164-4395-b665-39cae0e4386e","Type":"ContainerStarted","Data":"b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422"} Jul 19 15:20:01.743042 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:01.742513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" event={"ID":"c99cb155-0164-4395-b665-39cae0e4386e","Type":"ContainerStarted","Data":"bdba8ae425dbe3c70ff0829e0cd69ac8f00cb9225e9c44248f52e15f9079f3ac"} Jul 19 15:20:01.760690 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:01.760636 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" podStartSLOduration=1.760620076 podStartE2EDuration="1.760620076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:20:01.758477756 +0000 UTC m=+2614.180702094" watchObservedRunningTime="2026-07-19 15:20:01.760620076 +0000 UTC m=+2614.182844408" Jul 19 15:20:04.393310 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:04.393274 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:20:10.626067 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:10.626012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:10.626503 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:10.626090 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:10.638705 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:10.638674 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:10.802514 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:10.802479 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:20:31.402268 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:31.402232 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:20:50.699298 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:20:50.699256 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:21:06.003011 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:21:06.002976 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:21:25.902746 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:21:25.902659 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:21:28.298337 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:21:28.298307 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:21:28.310133 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:21:28.310109 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:21:45.633935 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:21:45.633899 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:22:02.996616 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:02.996580 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:22:18.498951 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:18.498917 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5"] Jul 19 15:22:19.419849 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:19.419815 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj"] Jul 19 15:22:21.993529 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:21.993494 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:22:33.518652 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.518615 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" podUID="cc2c0635-d4e6-41e7-93e0-31d361287ff2" containerName="main" containerID="cri-o://b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd" gracePeriod=45 Jul 19 15:22:33.773726 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.773665 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:22:33.820634 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.820604 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tmp-dir\") pod \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " Jul 19 15:22:33.820760 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.820641 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-model-cache\") pod \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " Jul 19 15:22:33.820760 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.820674 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-home\") pod \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " Jul 19 15:22:33.820760 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.820721 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ml66\" (UniqueName: \"kubernetes.io/projected/cc2c0635-d4e6-41e7-93e0-31d361287ff2-kube-api-access-2ml66\") pod \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " Jul 19 15:22:33.820760 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.820755 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tls-certs\") pod \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " Jul 19 15:22:33.821005 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.820776 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-dshm\") pod \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\" (UID: \"cc2c0635-d4e6-41e7-93e0-31d361287ff2\") " Jul 19 15:22:33.821005 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.820919 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc2c0635-d4e6-41e7-93e0-31d361287ff2" (UID: "cc2c0635-d4e6-41e7-93e0-31d361287ff2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:33.821005 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.820929 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc2c0635-d4e6-41e7-93e0-31d361287ff2" (UID: "cc2c0635-d4e6-41e7-93e0-31d361287ff2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:33.821155 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.821029 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:33.821155 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.821028 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-home" (OuterVolumeSpecName: "home") pod "cc2c0635-d4e6-41e7-93e0-31d361287ff2" (UID: "cc2c0635-d4e6-41e7-93e0-31d361287ff2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:33.821155 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.821042 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:33.823019 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.822985 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc2c0635-d4e6-41e7-93e0-31d361287ff2" (UID: "cc2c0635-d4e6-41e7-93e0-31d361287ff2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:22:33.823132 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.823016 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2c0635-d4e6-41e7-93e0-31d361287ff2-kube-api-access-2ml66" (OuterVolumeSpecName: "kube-api-access-2ml66") pod "cc2c0635-d4e6-41e7-93e0-31d361287ff2" (UID: "cc2c0635-d4e6-41e7-93e0-31d361287ff2"). InnerVolumeSpecName "kube-api-access-2ml66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:22:33.823132 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.823069 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-dshm" (OuterVolumeSpecName: "dshm") pod "cc2c0635-d4e6-41e7-93e0-31d361287ff2" (UID: "cc2c0635-d4e6-41e7-93e0-31d361287ff2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:33.921474 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.921444 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2c0635-d4e6-41e7-93e0-31d361287ff2-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:33.921474 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.921468 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:33.921474 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.921476 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2c0635-d4e6-41e7-93e0-31d361287ff2-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:33.921655 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:33.921483 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ml66\" (UniqueName: \"kubernetes.io/projected/cc2c0635-d4e6-41e7-93e0-31d361287ff2-kube-api-access-2ml66\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:34.441121 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.441011 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" podUID="c99cb155-0164-4395-b665-39cae0e4386e" containerName="main" containerID="cri-o://b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422" gracePeriod=45 Jul 19 15:22:34.529335 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.529300 2559 generic.go:356] "Generic (PLEG): container finished" podID="cc2c0635-d4e6-41e7-93e0-31d361287ff2" containerID="b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd" exitCode=0 Jul 19 15:22:34.529769 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.529346 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" Jul 19 15:22:34.529769 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.529351 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" event={"ID":"cc2c0635-d4e6-41e7-93e0-31d361287ff2","Type":"ContainerDied","Data":"b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd"} Jul 19 15:22:34.529769 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.529390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5" event={"ID":"cc2c0635-d4e6-41e7-93e0-31d361287ff2","Type":"ContainerDied","Data":"106c0cf92c5af0f9dc1f8f25560b3b17df54513ddffe6518361e5b6710f6cbf1"} Jul 19 15:22:34.529769 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.529415 2559 scope.go:117] "RemoveContainer" containerID="b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd" Jul 19 15:22:34.548982 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.548945 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5"] Jul 19 15:22:34.551117 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.551090 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/auth-test-model-7679e0-kserve-65ddb56695-6lkz5"] Jul 19 15:22:34.552989 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.552964 2559 scope.go:117] "RemoveContainer" containerID="b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd" Jul 19 15:22:34.553308 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:22:34.553285 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd\": container with ID starting with b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd not found: ID does not exist" containerID="b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd" Jul 19 15:22:34.553373 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.553319 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd"} err="failed to get container status \"b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd\": rpc error: code = NotFound desc = could not find container \"b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd\": container with ID starting with b542abfadc101163f94330b2cd8966f308a1cb0d4efb93879d9cb51cb27fd0cd not found: ID does not exist" Jul 19 15:22:34.689771 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.689745 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:22:34.729159 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729096 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-tmp-dir\") pod \"c99cb155-0164-4395-b665-39cae0e4386e\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " Jul 19 15:22:34.729159 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729143 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-dshm\") pod \"c99cb155-0164-4395-b665-39cae0e4386e\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " Jul 19 15:22:34.729332 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729186 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-home\") pod \"c99cb155-0164-4395-b665-39cae0e4386e\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " Jul 19 15:22:34.729332 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729212 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99cb155-0164-4395-b665-39cae0e4386e-tls-certs\") pod \"c99cb155-0164-4395-b665-39cae0e4386e\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " Jul 19 15:22:34.729332 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729254 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-model-cache\") pod \"c99cb155-0164-4395-b665-39cae0e4386e\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " Jul 19 15:22:34.729332 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729284 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scmgn\" (UniqueName: \"kubernetes.io/projected/c99cb155-0164-4395-b665-39cae0e4386e-kube-api-access-scmgn\") pod \"c99cb155-0164-4395-b665-39cae0e4386e\" (UID: \"c99cb155-0164-4395-b665-39cae0e4386e\") " Jul 19 15:22:34.729540 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729323 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c99cb155-0164-4395-b665-39cae0e4386e" (UID: "c99cb155-0164-4395-b665-39cae0e4386e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:34.729540 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729396 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-home" (OuterVolumeSpecName: "home") pod "c99cb155-0164-4395-b665-39cae0e4386e" (UID: "c99cb155-0164-4395-b665-39cae0e4386e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:34.729540 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729517 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c99cb155-0164-4395-b665-39cae0e4386e" (UID: "c99cb155-0164-4395-b665-39cae0e4386e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:34.729697 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729618 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:34.729697 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729637 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:34.729697 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.729650 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:34.731357 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.731339 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-dshm" (OuterVolumeSpecName: "dshm") pod "c99cb155-0164-4395-b665-39cae0e4386e" (UID: "c99cb155-0164-4395-b665-39cae0e4386e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:34.731563 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.731531 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99cb155-0164-4395-b665-39cae0e4386e-kube-api-access-scmgn" (OuterVolumeSpecName: "kube-api-access-scmgn") pod "c99cb155-0164-4395-b665-39cae0e4386e" (UID: "c99cb155-0164-4395-b665-39cae0e4386e"). InnerVolumeSpecName "kube-api-access-scmgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:22:34.731711 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.731695 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99cb155-0164-4395-b665-39cae0e4386e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c99cb155-0164-4395-b665-39cae0e4386e" (UID: "c99cb155-0164-4395-b665-39cae0e4386e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:22:34.830239 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.830208 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99cb155-0164-4395-b665-39cae0e4386e-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:34.830378 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.830251 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99cb155-0164-4395-b665-39cae0e4386e-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:34.830378 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:34.830262 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scmgn\" (UniqueName: \"kubernetes.io/projected/c99cb155-0164-4395-b665-39cae0e4386e-kube-api-access-scmgn\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:22:35.536421 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.536377 2559 generic.go:356] "Generic (PLEG): container finished" podID="c99cb155-0164-4395-b665-39cae0e4386e" containerID="b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422" exitCode=0 Jul 19 15:22:35.536824 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.536436 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" Jul 19 15:22:35.536824 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.536468 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" event={"ID":"c99cb155-0164-4395-b665-39cae0e4386e","Type":"ContainerDied","Data":"b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422"} Jul 19 15:22:35.536824 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.536513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj" event={"ID":"c99cb155-0164-4395-b665-39cae0e4386e","Type":"ContainerDied","Data":"bdba8ae425dbe3c70ff0829e0cd69ac8f00cb9225e9c44248f52e15f9079f3ac"} Jul 19 15:22:35.536824 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.536530 2559 scope.go:117] "RemoveContainer" containerID="b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422" Jul 19 15:22:35.546045 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.546026 2559 scope.go:117] "RemoveContainer" containerID="b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422" Jul 19 15:22:35.546342 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:22:35.546324 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422\": container with ID starting with b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422 not found: ID does not exist" containerID="b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422" Jul 19 15:22:35.546405 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.546352 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422"} err="failed to get container status \"b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422\": rpc error: code = NotFound desc = could not find container \"b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422\": container with ID starting with b706286ea15f594b5f360eb1f1a9a3327e58331b66804ca4140c55716bd98422 not found: ID does not exist" Jul 19 15:22:35.557179 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.557157 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj"] Jul 19 15:22:35.562661 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.562641 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/auth-test-model-3c4e71-kserve-5cbc8574f9-qwvxj"] Jul 19 15:22:35.893950 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:35.893920 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:22:36.107752 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:36.107710 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c99cb155-0164-4395-b665-39cae0e4386e" path="/var/lib/kubelet/pods/c99cb155-0164-4395-b665-39cae0e4386e/volumes" Jul 19 15:22:36.108211 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:22:36.108190 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2c0635-d4e6-41e7-93e0-31d361287ff2" path="/var/lib/kubelet/pods/cc2c0635-d4e6-41e7-93e0-31d361287ff2/volumes" Jul 19 15:23:00.704629 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.704075 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n"] Jul 19 15:23:00.708944 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.704843 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2c0635-d4e6-41e7-93e0-31d361287ff2" containerName="main" Jul 19 15:23:00.708944 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.704878 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2c0635-d4e6-41e7-93e0-31d361287ff2" containerName="main" Jul 19 15:23:00.708944 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.704934 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c99cb155-0164-4395-b665-39cae0e4386e" containerName="main" Jul 19 15:23:00.708944 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.704945 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c99cb155-0164-4395-b665-39cae0e4386e" containerName="main" Jul 19 15:23:00.708944 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.705123 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c99cb155-0164-4395-b665-39cae0e4386e" containerName="main" Jul 19 15:23:00.708944 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.705148 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2c0635-d4e6-41e7-93e0-31d361287ff2" containerName="main" Jul 19 15:23:00.709314 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.709060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.712326 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.712296 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"default-dockercfg-5zs5j\"" Jul 19 15:23:00.712459 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.712308 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"sub-test-model-3c4e71-kserve-self-signed-certs\"" Jul 19 15:23:00.712536 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.712498 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"kube-root-ca.crt\"" Jul 19 15:23:00.712593 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.712580 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"openshift-service-ca.crt\"" Jul 19 15:23:00.714245 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.714220 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n"] Jul 19 15:23:00.856781 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.856749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-model-cache\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.856959 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.856799 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tls-certs\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.856959 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.856820 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tmp-dir\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.856959 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.856857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fch5h\" (UniqueName: \"kubernetes.io/projected/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-kube-api-access-fch5h\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.857070 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.856974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-home\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.857070 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.857015 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-dshm\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.957838 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.957750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-model-cache\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.957838 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.957810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tls-certs\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.958053 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.957929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tmp-dir\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.958053 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.957965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fch5h\" (UniqueName: \"kubernetes.io/projected/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-kube-api-access-fch5h\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.958167 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.958119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-home\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.958222 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.958191 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-dshm\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.958276 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.958237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-model-cache\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.958276 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.958246 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tmp-dir\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.958422 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.958404 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-home\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.960258 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.960236 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-dshm\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.960403 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.960384 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tls-certs\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:00.965474 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:00.965453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fch5h\" (UniqueName: \"kubernetes.io/projected/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-kube-api-access-fch5h\") pod \"sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:01.019674 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:01.019644 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:01.147517 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:01.147494 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n"] Jul 19 15:23:01.148394 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:23:01.148371 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc1ec8eb_a6b0_4a90_8fff_c431d9651571.slice/crio-6dccb2f3e7383f995222cd10cdf00182f84891663add87c7d1ef5945d1565f63 WatchSource:0}: Error finding container 6dccb2f3e7383f995222cd10cdf00182f84891663add87c7d1ef5945d1565f63: Status 404 returned error can't find the container with id 6dccb2f3e7383f995222cd10cdf00182f84891663add87c7d1ef5945d1565f63 Jul 19 15:23:01.676899 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:01.676851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" event={"ID":"cc1ec8eb-a6b0-4a90-8fff-c431d9651571","Type":"ContainerStarted","Data":"6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd"} Jul 19 15:23:01.677081 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:01.676910 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" event={"ID":"cc1ec8eb-a6b0-4a90-8fff-c431d9651571","Type":"ContainerStarted","Data":"6dccb2f3e7383f995222cd10cdf00182f84891663add87c7d1ef5945d1565f63"} Jul 19 15:23:01.694960 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:01.694907 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" podStartSLOduration=1.6948919949999999 podStartE2EDuration="1.694891995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:23:01.692987192 +0000 UTC m=+2794.115211545" watchObservedRunningTime="2026-07-19 15:23:01.694891995 +0000 UTC m=+2794.117116326" Jul 19 15:23:04.810893 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:04.810836 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:23:11.020635 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:11.020600 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:11.020635 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:11.020646 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:11.033020 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:11.032996 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:11.736842 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:11.736814 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:23:29.697858 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:29.697820 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:23:43.595816 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:43.595779 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:23:51.604162 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:23:51.604124 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n"] Jul 19 15:24:01.904931 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:01.904897 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw"] Jul 19 15:24:01.908477 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:01.908459 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:01.910924 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:01.910901 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7679e0\"/\"default-dockercfg-dcp4j\"" Jul 19 15:24:01.912064 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:01.912036 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7679e0\"/\"rate-test-model-7679e0-kserve-self-signed-certs\"" Jul 19 15:24:01.912172 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:01.912047 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7679e0\"/\"kube-root-ca.crt\"" Jul 19 15:24:01.912172 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:01.912071 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7679e0\"/\"openshift-service-ca.crt\"" Jul 19 15:24:01.917681 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:01.917658 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw"] Jul 19 15:24:02.097984 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.097943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-tmp-dir\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.098191 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.098003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc295d-4625-469d-94d4-f12eab7d4c3a-tls-certs\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.098191 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.098028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkd5r\" (UniqueName: \"kubernetes.io/projected/41cc295d-4625-469d-94d4-f12eab7d4c3a-kube-api-access-mkd5r\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.098191 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.098077 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-dshm\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.098191 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.098102 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-model-cache\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.098191 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.098171 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-home\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.198699 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.198614 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-dshm\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.198699 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.198649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-model-cache\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.198699 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.198669 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-home\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.198995 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.198733 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-tmp-dir\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.198995 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.198786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc295d-4625-469d-94d4-f12eab7d4c3a-tls-certs\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.198995 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.198814 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkd5r\" (UniqueName: \"kubernetes.io/projected/41cc295d-4625-469d-94d4-f12eab7d4c3a-kube-api-access-mkd5r\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.199151 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.199135 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-model-cache\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.199223 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.199194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-home\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.199260 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.199228 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-tmp-dir\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.200878 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.200841 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-dshm\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.201182 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.201167 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc295d-4625-469d-94d4-f12eab7d4c3a-tls-certs\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.207252 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.207229 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkd5r\" (UniqueName: \"kubernetes.io/projected/41cc295d-4625-469d-94d4-f12eab7d4c3a-kube-api-access-mkd5r\") pod \"rate-test-model-7679e0-kserve-746c55f79-dvtqw\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.218635 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.218614 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:02.359403 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.359374 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw"] Jul 19 15:24:02.361247 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:24:02.361218 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41cc295d_4625_469d_94d4_f12eab7d4c3a.slice/crio-4e60b203ab98a179c39503a4945dbdc3fbb09f78c514e5550ceea592086b8c63 WatchSource:0}: Error finding container 4e60b203ab98a179c39503a4945dbdc3fbb09f78c514e5550ceea592086b8c63: Status 404 returned error can't find the container with id 4e60b203ab98a179c39503a4945dbdc3fbb09f78c514e5550ceea592086b8c63 Jul 19 15:24:02.900008 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.899970 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:24:02.976258 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.976222 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" event={"ID":"41cc295d-4625-469d-94d4-f12eab7d4c3a","Type":"ContainerStarted","Data":"9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05"} Jul 19 15:24:02.976627 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.976267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" event={"ID":"41cc295d-4625-469d-94d4-f12eab7d4c3a","Type":"ContainerStarted","Data":"4e60b203ab98a179c39503a4945dbdc3fbb09f78c514e5550ceea592086b8c63"} Jul 19 15:24:02.994687 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:02.994634 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" podStartSLOduration=1.9946141819999998 podStartE2EDuration="1.994614182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:24:02.994550431 +0000 UTC m=+2855.416774763" watchObservedRunningTime="2026-07-19 15:24:02.994614182 +0000 UTC m=+2855.416838512" Jul 19 15:24:06.624441 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.624374 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" podUID="cc1ec8eb-a6b0-4a90-8fff-c431d9651571" containerName="main" containerID="cri-o://6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd" gracePeriod=45 Jul 19 15:24:06.864030 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.864002 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:24:06.935292 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.935215 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fch5h\" (UniqueName: \"kubernetes.io/projected/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-kube-api-access-fch5h\") pod \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " Jul 19 15:24:06.935292 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.935257 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-model-cache\") pod \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " Jul 19 15:24:06.935292 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.935286 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tmp-dir\") pod \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " Jul 19 15:24:06.935577 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.935317 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tls-certs\") pod \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " Jul 19 15:24:06.935577 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.935411 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-dshm\") pod \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " Jul 19 15:24:06.935577 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.935534 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc1ec8eb-a6b0-4a90-8fff-c431d9651571" (UID: "cc1ec8eb-a6b0-4a90-8fff-c431d9651571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:24:06.935577 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.935552 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc1ec8eb-a6b0-4a90-8fff-c431d9651571" (UID: "cc1ec8eb-a6b0-4a90-8fff-c431d9651571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:24:06.935752 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.935741 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:24:06.935788 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.935772 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:24:06.937520 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.937493 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-kube-api-access-fch5h" (OuterVolumeSpecName: "kube-api-access-fch5h") pod "cc1ec8eb-a6b0-4a90-8fff-c431d9651571" (UID: "cc1ec8eb-a6b0-4a90-8fff-c431d9651571"). InnerVolumeSpecName "kube-api-access-fch5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:24:06.937625 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.937537 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc1ec8eb-a6b0-4a90-8fff-c431d9651571" (UID: "cc1ec8eb-a6b0-4a90-8fff-c431d9651571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:24:06.937625 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.937554 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-dshm" (OuterVolumeSpecName: "dshm") pod "cc1ec8eb-a6b0-4a90-8fff-c431d9651571" (UID: "cc1ec8eb-a6b0-4a90-8fff-c431d9651571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:24:06.997351 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.997325 2559 generic.go:356] "Generic (PLEG): container finished" podID="cc1ec8eb-a6b0-4a90-8fff-c431d9651571" containerID="6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd" exitCode=0 Jul 19 15:24:06.997488 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.997365 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" Jul 19 15:24:06.997488 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.997415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" event={"ID":"cc1ec8eb-a6b0-4a90-8fff-c431d9651571","Type":"ContainerDied","Data":"6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd"} Jul 19 15:24:06.997488 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.997456 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n" event={"ID":"cc1ec8eb-a6b0-4a90-8fff-c431d9651571","Type":"ContainerDied","Data":"6dccb2f3e7383f995222cd10cdf00182f84891663add87c7d1ef5945d1565f63"} Jul 19 15:24:06.997488 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:06.997473 2559 scope.go:117] "RemoveContainer" containerID="6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd" Jul 19 15:24:07.006077 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.006062 2559 scope.go:117] "RemoveContainer" containerID="6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd" Jul 19 15:24:07.006354 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:24:07.006337 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd\": container with ID starting with 6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd not found: ID does not exist" containerID="6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd" Jul 19 15:24:07.006416 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.006369 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd"} err="failed to get container status \"6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd\": rpc error: code = NotFound desc = could not find container \"6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd\": container with ID starting with 6ad660cb39965f27434c0f78c46f75d2d7ec36c2673a35bc7ff5c5c7d3b9bcfd not found: ID does not exist" Jul 19 15:24:07.036221 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.036197 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-home\") pod \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\" (UID: \"cc1ec8eb-a6b0-4a90-8fff-c431d9651571\") " Jul 19 15:24:07.036397 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.036385 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:24:07.036444 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.036400 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fch5h\" (UniqueName: \"kubernetes.io/projected/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-kube-api-access-fch5h\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:24:07.036444 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.036403 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-home" (OuterVolumeSpecName: "home") pod "cc1ec8eb-a6b0-4a90-8fff-c431d9651571" (UID: "cc1ec8eb-a6b0-4a90-8fff-c431d9651571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:24:07.036444 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.036410 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:24:07.137761 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.137716 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1ec8eb-a6b0-4a90-8fff-c431d9651571-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:24:07.322852 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.322821 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n"] Jul 19 15:24:07.326304 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:07.326278 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/sub-test-model-3c4e71-kserve-c6c8c9477-2qr9n"] Jul 19 15:24:08.108737 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:08.108702 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc1ec8eb-a6b0-4a90-8fff-c431d9651571" path="/var/lib/kubelet/pods/cc1ec8eb-a6b0-4a90-8fff-c431d9651571/volumes" Jul 19 15:24:12.218949 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:12.218911 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:12.218949 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:12.218953 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:12.231108 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:12.231082 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:12.491946 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:12.491841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:24:13.042167 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:13.042111 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:24:21.703521 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.703450 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj"] Jul 19 15:24:21.703855 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.703773 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc1ec8eb-a6b0-4a90-8fff-c431d9651571" containerName="main" Jul 19 15:24:21.703855 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.703784 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc1ec8eb-a6b0-4a90-8fff-c431d9651571" containerName="main" Jul 19 15:24:21.703855 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.703852 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc1ec8eb-a6b0-4a90-8fff-c431d9651571" containerName="main" Jul 19 15:24:21.706991 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.706973 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.710374 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.710351 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"rate-test-model-3c4e71-kserve-self-signed-certs\"" Jul 19 15:24:21.710511 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.710389 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"default-dockercfg-5zs5j\"" Jul 19 15:24:21.710511 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.710355 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"kube-root-ca.crt\"" Jul 19 15:24:21.710511 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.710355 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c4e71\"/\"openshift-service-ca.crt\"" Jul 19 15:24:21.714962 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.714942 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj"] Jul 19 15:24:21.769129 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.769098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-model-cache\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.769252 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.769134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-dshm\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.769252 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.769205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-home\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.769345 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.769255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6394b69b-9527-41e4-bf08-f1ea860b7458-tls-certs\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.769345 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.769289 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwdff\" (UniqueName: \"kubernetes.io/projected/6394b69b-9527-41e4-bf08-f1ea860b7458-kube-api-access-zwdff\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.769345 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.769310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-tmp-dir\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.870239 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.870206 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-model-cache\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.870397 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.870242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-dshm\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.870397 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.870302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-home\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.870397 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.870348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6394b69b-9527-41e4-bf08-f1ea860b7458-tls-certs\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.870397 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.870390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwdff\" (UniqueName: \"kubernetes.io/projected/6394b69b-9527-41e4-bf08-f1ea860b7458-kube-api-access-zwdff\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.870606 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.870420 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-tmp-dir\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.870694 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.870663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-model-cache\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.870767 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.870751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-home\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.870827 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.870782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-tmp-dir\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.872521 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.872500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-dshm\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.872925 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.872903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6394b69b-9527-41e4-bf08-f1ea860b7458-tls-certs\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:21.877531 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:21.877507 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwdff\" (UniqueName: \"kubernetes.io/projected/6394b69b-9527-41e4-bf08-f1ea860b7458-kube-api-access-zwdff\") pod \"rate-test-model-3c4e71-kserve-567568947f-w6lpj\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:22.016516 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:22.016451 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:22.147363 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:22.147335 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj"] Jul 19 15:24:22.149448 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:24:22.149420 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6394b69b_9527_41e4_bf08_f1ea860b7458.slice/crio-13650317fa403ec1d8ec2f9fc685c8657b70ecf7fd7a856c90d8cfa3bc890f36 WatchSource:0}: Error finding container 13650317fa403ec1d8ec2f9fc685c8657b70ecf7fd7a856c90d8cfa3bc890f36: Status 404 returned error can't find the container with id 13650317fa403ec1d8ec2f9fc685c8657b70ecf7fd7a856c90d8cfa3bc890f36 Jul 19 15:24:23.088327 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:23.088289 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" event={"ID":"6394b69b-9527-41e4-bf08-f1ea860b7458","Type":"ContainerStarted","Data":"a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704"} Jul 19 15:24:23.088776 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:23.088335 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" event={"ID":"6394b69b-9527-41e4-bf08-f1ea860b7458","Type":"ContainerStarted","Data":"13650317fa403ec1d8ec2f9fc685c8657b70ecf7fd7a856c90d8cfa3bc890f36"} Jul 19 15:24:23.104067 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:23.104013 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" podStartSLOduration=2.10399536 podStartE2EDuration="2.10399536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:24:23.103562783 +0000 UTC m=+2875.525787114" watchObservedRunningTime="2026-07-19 15:24:23.10399536 +0000 UTC m=+2875.526219690" Jul 19 15:24:25.791815 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:25.791780 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:24:32.017585 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:32.017548 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:32.018072 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:32.017603 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:32.029744 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:32.029719 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:32.145844 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:32.145810 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:24:39.292997 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:39.292962 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:24:54.494384 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:24:54.494343 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:25:02.428349 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:02.428314 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw"] Jul 19 15:25:03.415702 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:03.415668 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj"] Jul 19 15:25:04.401802 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.401772 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7292-140663706024496"] Jul 19 15:25:04.405365 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.405347 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663706024496" Jul 19 15:25:04.407935 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.407919 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9sx9t\"" Jul 19 15:25:04.411749 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.411727 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-140663706024496"] Jul 19 15:25:04.453394 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.453368 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8cn2\" (UniqueName: \"kubernetes.io/projected/0893d646-3058-461a-bc36-ac7e2941aa12-kube-api-access-t8cn2\") pod \"test-curl-7292-140663706024496\" (UID: \"0893d646-3058-461a-bc36-ac7e2941aa12\") " pod="opendatahub/test-curl-7292-140663706024496" Jul 19 15:25:04.554835 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.554800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8cn2\" (UniqueName: \"kubernetes.io/projected/0893d646-3058-461a-bc36-ac7e2941aa12-kube-api-access-t8cn2\") pod \"test-curl-7292-140663706024496\" (UID: \"0893d646-3058-461a-bc36-ac7e2941aa12\") " pod="opendatahub/test-curl-7292-140663706024496" Jul 19 15:25:04.562879 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.562826 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8cn2\" (UniqueName: \"kubernetes.io/projected/0893d646-3058-461a-bc36-ac7e2941aa12-kube-api-access-t8cn2\") pod \"test-curl-7292-140663706024496\" (UID: \"0893d646-3058-461a-bc36-ac7e2941aa12\") " pod="opendatahub/test-curl-7292-140663706024496" Jul 19 15:25:04.715937 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.715816 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663706024496" Jul 19 15:25:04.855608 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.855583 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-140663706024496"] Jul 19 15:25:04.857236 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:25:04.857207 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0893d646_3058_461a_bc36_ac7e2941aa12.slice/crio-d947ee1bedee5b4a3972dc807aa77fdd6ca0077e9a4dcb0c036d3bd6fca49355 WatchSource:0}: Error finding container d947ee1bedee5b4a3972dc807aa77fdd6ca0077e9a4dcb0c036d3bd6fca49355: Status 404 returned error can't find the container with id d947ee1bedee5b4a3972dc807aa77fdd6ca0077e9a4dcb0c036d3bd6fca49355 Jul 19 15:25:04.858808 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:04.858790 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:25:05.326181 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:05.326118 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-140663706024496" event={"ID":"0893d646-3058-461a-bc36-ac7e2941aa12","Type":"ContainerStarted","Data":"d947ee1bedee5b4a3972dc807aa77fdd6ca0077e9a4dcb0c036d3bd6fca49355"} Jul 19 15:25:06.332217 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:06.332183 2559 generic.go:356] "Generic (PLEG): container finished" podID="0893d646-3058-461a-bc36-ac7e2941aa12" containerID="a87b98d55be7a538a8cf0d79cfb031b4cb3951ce47ee2680d96883ce44390569" exitCode=0 Jul 19 15:25:06.332607 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:06.332257 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-140663706024496" event={"ID":"0893d646-3058-461a-bc36-ac7e2941aa12","Type":"ContainerDied","Data":"a87b98d55be7a538a8cf0d79cfb031b4cb3951ce47ee2680d96883ce44390569"} Jul 19 15:25:07.468093 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:07.468063 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663706024496" Jul 19 15:25:07.501750 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:07.501725 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7292-140663706024496_0893d646-3058-461a-bc36-ac7e2941aa12/test-curl-7292-140663706024496/0.log" Jul 19 15:25:07.526509 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:07.526474 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7292-140663706024496"] Jul 19 15:25:07.530376 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:07.530339 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7292-140663706024496"] Jul 19 15:25:07.581518 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:07.581484 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8cn2\" (UniqueName: \"kubernetes.io/projected/0893d646-3058-461a-bc36-ac7e2941aa12-kube-api-access-t8cn2\") pod \"0893d646-3058-461a-bc36-ac7e2941aa12\" (UID: \"0893d646-3058-461a-bc36-ac7e2941aa12\") " Jul 19 15:25:07.583706 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:07.583677 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0893d646-3058-461a-bc36-ac7e2941aa12-kube-api-access-t8cn2" (OuterVolumeSpecName: "kube-api-access-t8cn2") pod "0893d646-3058-461a-bc36-ac7e2941aa12" (UID: "0893d646-3058-461a-bc36-ac7e2941aa12"). InnerVolumeSpecName "kube-api-access-t8cn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:25:07.682433 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:07.682396 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8cn2\" (UniqueName: \"kubernetes.io/projected/0893d646-3058-461a-bc36-ac7e2941aa12-kube-api-access-t8cn2\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:08.108580 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:08.108547 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0893d646-3058-461a-bc36-ac7e2941aa12" path="/var/lib/kubelet/pods/0893d646-3058-461a-bc36-ac7e2941aa12/volumes" Jul 19 15:25:08.344499 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:08.344469 2559 scope.go:117] "RemoveContainer" containerID="a87b98d55be7a538a8cf0d79cfb031b4cb3951ce47ee2680d96883ce44390569" Jul 19 15:25:08.344499 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:08.344494 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663706024496" Jul 19 15:25:09.789192 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:09.789158 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:25:10.693427 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.693394 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:10.693738 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.693725 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0893d646-3058-461a-bc36-ac7e2941aa12" containerName="test-curl-7292-140663706024496" Jul 19 15:25:10.693781 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.693739 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0893d646-3058-461a-bc36-ac7e2941aa12" containerName="test-curl-7292-140663706024496" Jul 19 15:25:10.693814 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.693807 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0893d646-3058-461a-bc36-ac7e2941aa12" containerName="test-curl-7292-140663706024496" Jul 19 15:25:10.696904 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.696888 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:10.699061 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.699042 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9sx9t\"" Jul 19 15:25:10.702604 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.702579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:10.810629 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.810588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9245l\" (UniqueName: \"kubernetes.io/projected/e7ac1c9d-2826-483a-84ed-2eb71a417ce2-kube-api-access-9245l\") pod \"test-curl-7292-140663705905312\" (UID: \"e7ac1c9d-2826-483a-84ed-2eb71a417ce2\") " pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:10.911758 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.911724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9245l\" (UniqueName: \"kubernetes.io/projected/e7ac1c9d-2826-483a-84ed-2eb71a417ce2-kube-api-access-9245l\") pod \"test-curl-7292-140663705905312\" (UID: \"e7ac1c9d-2826-483a-84ed-2eb71a417ce2\") " pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:10.919186 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:10.919161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9245l\" (UniqueName: \"kubernetes.io/projected/e7ac1c9d-2826-483a-84ed-2eb71a417ce2-kube-api-access-9245l\") pod \"test-curl-7292-140663705905312\" (UID: \"e7ac1c9d-2826-483a-84ed-2eb71a417ce2\") " pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:11.007395 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:11.007306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:11.135107 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:11.135079 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:11.136845 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:25:11.136814 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7ac1c9d_2826_483a_84ed_2eb71a417ce2.slice/crio-8aca725dceef4a7281aaea35fc8bd9b1ba441f4b48009a1f277d39f6da25e72f WatchSource:0}: Error finding container 8aca725dceef4a7281aaea35fc8bd9b1ba441f4b48009a1f277d39f6da25e72f: Status 404 returned error can't find the container with id 8aca725dceef4a7281aaea35fc8bd9b1ba441f4b48009a1f277d39f6da25e72f Jul 19 15:25:11.361589 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:11.361556 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-140663705905312" event={"ID":"e7ac1c9d-2826-483a-84ed-2eb71a417ce2","Type":"ContainerStarted","Data":"8aca725dceef4a7281aaea35fc8bd9b1ba441f4b48009a1f277d39f6da25e72f"} Jul 19 15:25:12.369096 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:12.369059 2559 generic.go:356] "Generic (PLEG): container finished" podID="e7ac1c9d-2826-483a-84ed-2eb71a417ce2" containerID="53d41a7ed02916260136b4dfe8be893515e84bf7b651d077d49e7dfb6a2f47ee" exitCode=0 Jul 19 15:25:12.369466 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:12.369150 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-140663705905312" event={"ID":"e7ac1c9d-2826-483a-84ed-2eb71a417ce2","Type":"ContainerDied","Data":"53d41a7ed02916260136b4dfe8be893515e84bf7b651d077d49e7dfb6a2f47ee"} Jul 19 15:25:13.508209 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.508185 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:13.546454 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.546433 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7292-140663705905312_e7ac1c9d-2826-483a-84ed-2eb71a417ce2/test-curl-7292-140663705905312/0.log" Jul 19 15:25:13.580294 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.580255 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:13.586296 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.586270 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:13.635745 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.635657 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9245l\" (UniqueName: \"kubernetes.io/projected/e7ac1c9d-2826-483a-84ed-2eb71a417ce2-kube-api-access-9245l\") pod \"e7ac1c9d-2826-483a-84ed-2eb71a417ce2\" (UID: \"e7ac1c9d-2826-483a-84ed-2eb71a417ce2\") " Jul 19 15:25:13.637718 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.637686 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ac1c9d-2826-483a-84ed-2eb71a417ce2-kube-api-access-9245l" (OuterVolumeSpecName: "kube-api-access-9245l") pod "e7ac1c9d-2826-483a-84ed-2eb71a417ce2" (UID: "e7ac1c9d-2826-483a-84ed-2eb71a417ce2"). InnerVolumeSpecName "kube-api-access-9245l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:25:13.709305 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.709271 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:13.709631 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.709618 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ac1c9d-2826-483a-84ed-2eb71a417ce2" containerName="test-curl-7292-140663705905312" Jul 19 15:25:13.709678 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.709633 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ac1c9d-2826-483a-84ed-2eb71a417ce2" containerName="test-curl-7292-140663705905312" Jul 19 15:25:13.709717 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.709705 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ac1c9d-2826-483a-84ed-2eb71a417ce2" containerName="test-curl-7292-140663705905312" Jul 19 15:25:13.712792 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.712776 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:13.719929 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.719900 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:13.737307 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.737281 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9245l\" (UniqueName: \"kubernetes.io/projected/e7ac1c9d-2826-483a-84ed-2eb71a417ce2-kube-api-access-9245l\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:13.838518 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.838487 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqs5\" (UniqueName: \"kubernetes.io/projected/f1945220-6ae5-45e2-b831-856c459d0b11-kube-api-access-8wqs5\") pod \"test-curl-7292-140663705905312\" (UID: \"f1945220-6ae5-45e2-b831-856c459d0b11\") " pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:13.940151 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.940044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqs5\" (UniqueName: \"kubernetes.io/projected/f1945220-6ae5-45e2-b831-856c459d0b11-kube-api-access-8wqs5\") pod \"test-curl-7292-140663705905312\" (UID: \"f1945220-6ae5-45e2-b831-856c459d0b11\") " pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:13.948170 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:13.948147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqs5\" (UniqueName: \"kubernetes.io/projected/f1945220-6ae5-45e2-b831-856c459d0b11-kube-api-access-8wqs5\") pod \"test-curl-7292-140663705905312\" (UID: \"f1945220-6ae5-45e2-b831-856c459d0b11\") " pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:14.024350 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:14.024316 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:14.109935 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:14.109902 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7ac1c9d-2826-483a-84ed-2eb71a417ce2" path="/var/lib/kubelet/pods/e7ac1c9d-2826-483a-84ed-2eb71a417ce2/volumes" Jul 19 15:25:14.153097 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:14.153067 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:14.154840 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:25:14.154813 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1945220_6ae5_45e2_b831_856c459d0b11.slice/crio-24d0367828ea71d3df799af85ee8b7c5a16afd77f374a95ad6dc812a309566bf WatchSource:0}: Error finding container 24d0367828ea71d3df799af85ee8b7c5a16afd77f374a95ad6dc812a309566bf: Status 404 returned error can't find the container with id 24d0367828ea71d3df799af85ee8b7c5a16afd77f374a95ad6dc812a309566bf Jul 19 15:25:14.380675 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:14.380642 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:14.380898 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:14.380678 2559 scope.go:117] "RemoveContainer" containerID="53d41a7ed02916260136b4dfe8be893515e84bf7b651d077d49e7dfb6a2f47ee" Jul 19 15:25:14.382002 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:14.381894 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-140663705905312" event={"ID":"f1945220-6ae5-45e2-b831-856c459d0b11","Type":"ContainerStarted","Data":"24d0367828ea71d3df799af85ee8b7c5a16afd77f374a95ad6dc812a309566bf"} Jul 19 15:25:15.389122 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:15.389086 2559 generic.go:356] "Generic (PLEG): container finished" podID="f1945220-6ae5-45e2-b831-856c459d0b11" containerID="c8162e9a455fd7b6da8b5d7034fa5683582eda9e291b20ffb8de6fddb1e361ba" exitCode=0 Jul 19 15:25:15.389122 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:15.389124 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-140663705905312" event={"ID":"f1945220-6ae5-45e2-b831-856c459d0b11","Type":"ContainerDied","Data":"c8162e9a455fd7b6da8b5d7034fa5683582eda9e291b20ffb8de6fddb1e361ba"} Jul 19 15:25:16.521786 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:16.521760 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:16.541223 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:16.541200 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7292-140663705905312_f1945220-6ae5-45e2-b831-856c459d0b11/test-curl-7292-140663705905312/0.log" Jul 19 15:25:16.563937 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:16.563897 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wqs5\" (UniqueName: \"kubernetes.io/projected/f1945220-6ae5-45e2-b831-856c459d0b11-kube-api-access-8wqs5\") pod \"f1945220-6ae5-45e2-b831-856c459d0b11\" (UID: \"f1945220-6ae5-45e2-b831-856c459d0b11\") " Jul 19 15:25:16.565047 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:16.565024 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:16.565818 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:16.565797 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1945220-6ae5-45e2-b831-856c459d0b11-kube-api-access-8wqs5" (OuterVolumeSpecName: "kube-api-access-8wqs5") pod "f1945220-6ae5-45e2-b831-856c459d0b11" (UID: "f1945220-6ae5-45e2-b831-856c459d0b11"). InnerVolumeSpecName "kube-api-access-8wqs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:25:16.570857 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:16.570837 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7292-140663705905312"] Jul 19 15:25:16.664741 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:16.664689 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wqs5\" (UniqueName: \"kubernetes.io/projected/f1945220-6ae5-45e2-b831-856c459d0b11-kube-api-access-8wqs5\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:17.401493 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.401467 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24d0367828ea71d3df799af85ee8b7c5a16afd77f374a95ad6dc812a309566bf" Jul 19 15:25:17.401493 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.401483 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:17.411634 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:25:17.411609 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7292-140663705905312\" is forbidden: User \"system:node:ip-10-0-134-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-156.ec2.internal' and this object" podUID="f1945220-6ae5-45e2-b831-856c459d0b11" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:17.448407 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.448380 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" podUID="41cc295d-4625-469d-94d4-f12eab7d4c3a" containerName="main" containerID="cri-o://9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05" gracePeriod=45 Jul 19 15:25:17.696559 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.696536 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:25:17.700065 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:25:17.700036 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7292-140663705905312\" is forbidden: User \"system:node:ip-10-0-134-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-156.ec2.internal' and this object" podUID="f1945220-6ae5-45e2-b831-856c459d0b11" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:17.772143 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772110 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-dshm\") pod \"41cc295d-4625-469d-94d4-f12eab7d4c3a\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " Jul 19 15:25:17.772259 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772161 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc295d-4625-469d-94d4-f12eab7d4c3a-tls-certs\") pod \"41cc295d-4625-469d-94d4-f12eab7d4c3a\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " Jul 19 15:25:17.772259 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772205 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkd5r\" (UniqueName: \"kubernetes.io/projected/41cc295d-4625-469d-94d4-f12eab7d4c3a-kube-api-access-mkd5r\") pod \"41cc295d-4625-469d-94d4-f12eab7d4c3a\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " Jul 19 15:25:17.772259 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772231 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-tmp-dir\") pod \"41cc295d-4625-469d-94d4-f12eab7d4c3a\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " Jul 19 15:25:17.772259 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772247 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-home\") pod \"41cc295d-4625-469d-94d4-f12eab7d4c3a\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " Jul 19 15:25:17.772522 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772297 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-model-cache\") pod \"41cc295d-4625-469d-94d4-f12eab7d4c3a\" (UID: \"41cc295d-4625-469d-94d4-f12eab7d4c3a\") " Jul 19 15:25:17.772578 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772519 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41cc295d-4625-469d-94d4-f12eab7d4c3a" (UID: "41cc295d-4625-469d-94d4-f12eab7d4c3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:25:17.772578 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772530 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-home" (OuterVolumeSpecName: "home") pod "41cc295d-4625-469d-94d4-f12eab7d4c3a" (UID: "41cc295d-4625-469d-94d4-f12eab7d4c3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:25:17.772674 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772591 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-model-cache" (OuterVolumeSpecName: "model-cache") pod "41cc295d-4625-469d-94d4-f12eab7d4c3a" (UID: "41cc295d-4625-469d-94d4-f12eab7d4c3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:25:17.772723 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772672 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:17.772723 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.772691 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:17.774336 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.774308 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-dshm" (OuterVolumeSpecName: "dshm") pod "41cc295d-4625-469d-94d4-f12eab7d4c3a" (UID: "41cc295d-4625-469d-94d4-f12eab7d4c3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:25:17.774456 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.774403 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cc295d-4625-469d-94d4-f12eab7d4c3a-kube-api-access-mkd5r" (OuterVolumeSpecName: "kube-api-access-mkd5r") pod "41cc295d-4625-469d-94d4-f12eab7d4c3a" (UID: "41cc295d-4625-469d-94d4-f12eab7d4c3a"). InnerVolumeSpecName "kube-api-access-mkd5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:25:17.774456 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.774411 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41cc295d-4625-469d-94d4-f12eab7d4c3a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41cc295d-4625-469d-94d4-f12eab7d4c3a" (UID: "41cc295d-4625-469d-94d4-f12eab7d4c3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:25:17.874113 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.874077 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:17.874113 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.874112 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc295d-4625-469d-94d4-f12eab7d4c3a-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:17.874286 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.874140 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc295d-4625-469d-94d4-f12eab7d4c3a-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:17.874286 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:17.874150 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkd5r\" (UniqueName: \"kubernetes.io/projected/41cc295d-4625-469d-94d4-f12eab7d4c3a-kube-api-access-mkd5r\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:18.107207 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:25:18.107174 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7292-140663705905312\" is forbidden: User \"system:node:ip-10-0-134-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-156.ec2.internal' and this object" podUID="f1945220-6ae5-45e2-b831-856c459d0b11" pod="opendatahub/test-curl-7292-140663705905312" Jul 19 15:25:18.107387 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.107353 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1945220-6ae5-45e2-b831-856c459d0b11" path="/var/lib/kubelet/pods/f1945220-6ae5-45e2-b831-856c459d0b11/volumes" Jul 19 15:25:18.414682 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.414599 2559 generic.go:356] "Generic (PLEG): container finished" podID="41cc295d-4625-469d-94d4-f12eab7d4c3a" containerID="9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05" exitCode=0 Jul 19 15:25:18.414682 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.414653 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" Jul 19 15:25:18.414896 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.414677 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" event={"ID":"41cc295d-4625-469d-94d4-f12eab7d4c3a","Type":"ContainerDied","Data":"9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05"} Jul 19 15:25:18.414896 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.414724 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw" event={"ID":"41cc295d-4625-469d-94d4-f12eab7d4c3a","Type":"ContainerDied","Data":"4e60b203ab98a179c39503a4945dbdc3fbb09f78c514e5550ceea592086b8c63"} Jul 19 15:25:18.414896 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.414746 2559 scope.go:117] "RemoveContainer" containerID="9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05" Jul 19 15:25:18.424280 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.424256 2559 scope.go:117] "RemoveContainer" containerID="9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05" Jul 19 15:25:18.424602 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:25:18.424578 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05\": container with ID starting with 9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05 not found: ID does not exist" containerID="9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05" Jul 19 15:25:18.424698 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.424609 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05"} err="failed to get container status \"9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05\": rpc error: code = NotFound desc = could not find container \"9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05\": container with ID starting with 9e60edf3719035fa6e866100d89099e8c9b74f7ca9b49c61ec8dbabff610ff05 not found: ID does not exist" Jul 19 15:25:18.435736 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.435709 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" podUID="6394b69b-9527-41e4-bf08-f1ea860b7458" containerName="main" containerID="cri-o://a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704" gracePeriod=45 Jul 19 15:25:18.446852 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.446828 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw"] Jul 19 15:25:18.450673 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.450652 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7679e0/rate-test-model-7679e0-kserve-746c55f79-dvtqw"] Jul 19 15:25:18.672073 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.672015 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:25:18.783624 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.783589 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-home\") pod \"6394b69b-9527-41e4-bf08-f1ea860b7458\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " Jul 19 15:25:18.784054 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.783659 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-dshm\") pod \"6394b69b-9527-41e4-bf08-f1ea860b7458\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " Jul 19 15:25:18.784054 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.783694 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-tmp-dir\") pod \"6394b69b-9527-41e4-bf08-f1ea860b7458\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " Jul 19 15:25:18.784054 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.783746 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwdff\" (UniqueName: \"kubernetes.io/projected/6394b69b-9527-41e4-bf08-f1ea860b7458-kube-api-access-zwdff\") pod \"6394b69b-9527-41e4-bf08-f1ea860b7458\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " Jul 19 15:25:18.784054 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.783794 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-model-cache\") pod \"6394b69b-9527-41e4-bf08-f1ea860b7458\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " Jul 19 15:25:18.784054 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.783826 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6394b69b-9527-41e4-bf08-f1ea860b7458-tls-certs\") pod \"6394b69b-9527-41e4-bf08-f1ea860b7458\" (UID: \"6394b69b-9527-41e4-bf08-f1ea860b7458\") " Jul 19 15:25:18.784054 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.783938 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-home" (OuterVolumeSpecName: "home") pod "6394b69b-9527-41e4-bf08-f1ea860b7458" (UID: "6394b69b-9527-41e4-bf08-f1ea860b7458"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:25:18.784054 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.783978 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6394b69b-9527-41e4-bf08-f1ea860b7458" (UID: "6394b69b-9527-41e4-bf08-f1ea860b7458"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:25:18.784406 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.784081 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-model-cache" (OuterVolumeSpecName: "model-cache") pod "6394b69b-9527-41e4-bf08-f1ea860b7458" (UID: "6394b69b-9527-41e4-bf08-f1ea860b7458"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:25:18.784406 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.784208 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:18.784406 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.784227 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:18.784406 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.784235 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:18.785770 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.785744 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-dshm" (OuterVolumeSpecName: "dshm") pod "6394b69b-9527-41e4-bf08-f1ea860b7458" (UID: "6394b69b-9527-41e4-bf08-f1ea860b7458"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:25:18.785908 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.785812 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6394b69b-9527-41e4-bf08-f1ea860b7458-kube-api-access-zwdff" (OuterVolumeSpecName: "kube-api-access-zwdff") pod "6394b69b-9527-41e4-bf08-f1ea860b7458" (UID: "6394b69b-9527-41e4-bf08-f1ea860b7458"). InnerVolumeSpecName "kube-api-access-zwdff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:25:18.785973 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.785951 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6394b69b-9527-41e4-bf08-f1ea860b7458-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6394b69b-9527-41e4-bf08-f1ea860b7458" (UID: "6394b69b-9527-41e4-bf08-f1ea860b7458"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:25:18.884742 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.884710 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwdff\" (UniqueName: \"kubernetes.io/projected/6394b69b-9527-41e4-bf08-f1ea860b7458-kube-api-access-zwdff\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:18.884742 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.884734 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6394b69b-9527-41e4-bf08-f1ea860b7458-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:18.884742 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:18.884747 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6394b69b-9527-41e4-bf08-f1ea860b7458-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:19.421947 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.421916 2559 generic.go:356] "Generic (PLEG): container finished" podID="6394b69b-9527-41e4-bf08-f1ea860b7458" containerID="a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704" exitCode=0 Jul 19 15:25:19.422154 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.421956 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" Jul 19 15:25:19.422154 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.421976 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" event={"ID":"6394b69b-9527-41e4-bf08-f1ea860b7458","Type":"ContainerDied","Data":"a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704"} Jul 19 15:25:19.422154 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.422006 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj" event={"ID":"6394b69b-9527-41e4-bf08-f1ea860b7458","Type":"ContainerDied","Data":"13650317fa403ec1d8ec2f9fc685c8657b70ecf7fd7a856c90d8cfa3bc890f36"} Jul 19 15:25:19.422154 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.422022 2559 scope.go:117] "RemoveContainer" containerID="a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704" Jul 19 15:25:19.433075 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.433044 2559 scope.go:117] "RemoveContainer" containerID="a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704" Jul 19 15:25:19.433582 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:25:19.433561 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704\": container with ID starting with a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704 not found: ID does not exist" containerID="a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704" Jul 19 15:25:19.433643 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.433591 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704"} err="failed to get container status \"a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704\": rpc error: code = NotFound desc = could not find container \"a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704\": container with ID starting with a8853c1e1d67be12fc2105a9540fceb1bde90bf0cb60c85a2d27e026f4cee704 not found: ID does not exist" Jul 19 15:25:19.444411 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.444387 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj"] Jul 19 15:25:19.448767 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.448747 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3c4e71/rate-test-model-3c4e71-kserve-567568947f-w6lpj"] Jul 19 15:25:19.748315 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748228 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7292-140663705906320"] Jul 19 15:25:19.748660 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748641 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6394b69b-9527-41e4-bf08-f1ea860b7458" containerName="main" Jul 19 15:25:19.748747 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748662 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6394b69b-9527-41e4-bf08-f1ea860b7458" containerName="main" Jul 19 15:25:19.748747 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748699 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41cc295d-4625-469d-94d4-f12eab7d4c3a" containerName="main" Jul 19 15:25:19.748747 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748708 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="41cc295d-4625-469d-94d4-f12eab7d4c3a" containerName="main" Jul 19 15:25:19.748747 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748718 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1945220-6ae5-45e2-b831-856c459d0b11" containerName="test-curl-7292-140663705905312" Jul 19 15:25:19.748747 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748728 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1945220-6ae5-45e2-b831-856c459d0b11" containerName="test-curl-7292-140663705905312" Jul 19 15:25:19.749001 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748813 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="41cc295d-4625-469d-94d4-f12eab7d4c3a" containerName="main" Jul 19 15:25:19.749001 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748826 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1945220-6ae5-45e2-b831-856c459d0b11" containerName="test-curl-7292-140663705905312" Jul 19 15:25:19.749001 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.748838 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6394b69b-9527-41e4-bf08-f1ea860b7458" containerName="main" Jul 19 15:25:19.753142 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.753122 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705906320" Jul 19 15:25:19.756155 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.756134 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9sx9t\"" Jul 19 15:25:19.758713 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.758693 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-140663705906320"] Jul 19 15:25:19.891658 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.891627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2l4\" (UniqueName: \"kubernetes.io/projected/9835bbee-48a4-4623-ba54-9cca570c4b6f-kube-api-access-kv2l4\") pod \"test-curl-7292-140663705906320\" (UID: \"9835bbee-48a4-4623-ba54-9cca570c4b6f\") " pod="opendatahub/test-curl-7292-140663705906320" Jul 19 15:25:19.992769 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:19.992734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2l4\" (UniqueName: \"kubernetes.io/projected/9835bbee-48a4-4623-ba54-9cca570c4b6f-kube-api-access-kv2l4\") pod \"test-curl-7292-140663705906320\" (UID: \"9835bbee-48a4-4623-ba54-9cca570c4b6f\") " pod="opendatahub/test-curl-7292-140663705906320" Jul 19 15:25:20.003037 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:20.002975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2l4\" (UniqueName: \"kubernetes.io/projected/9835bbee-48a4-4623-ba54-9cca570c4b6f-kube-api-access-kv2l4\") pod \"test-curl-7292-140663705906320\" (UID: \"9835bbee-48a4-4623-ba54-9cca570c4b6f\") " pod="opendatahub/test-curl-7292-140663705906320" Jul 19 15:25:20.063430 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:20.063398 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705906320" Jul 19 15:25:20.108016 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:20.107986 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41cc295d-4625-469d-94d4-f12eab7d4c3a" path="/var/lib/kubelet/pods/41cc295d-4625-469d-94d4-f12eab7d4c3a/volumes" Jul 19 15:25:20.108382 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:20.108370 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6394b69b-9527-41e4-bf08-f1ea860b7458" path="/var/lib/kubelet/pods/6394b69b-9527-41e4-bf08-f1ea860b7458/volumes" Jul 19 15:25:20.199566 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:20.199542 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-140663705906320"] Jul 19 15:25:20.200986 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:25:20.200958 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9835bbee_48a4_4623_ba54_9cca570c4b6f.slice/crio-d4d3282d1f0d3afedef81f77bd925df42a765b15bddbdd7b8fb0abc903e9676e WatchSource:0}: Error finding container d4d3282d1f0d3afedef81f77bd925df42a765b15bddbdd7b8fb0abc903e9676e: Status 404 returned error can't find the container with id d4d3282d1f0d3afedef81f77bd925df42a765b15bddbdd7b8fb0abc903e9676e Jul 19 15:25:20.428982 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:20.428950 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-140663705906320" event={"ID":"9835bbee-48a4-4623-ba54-9cca570c4b6f","Type":"ContainerStarted","Data":"d4d3282d1f0d3afedef81f77bd925df42a765b15bddbdd7b8fb0abc903e9676e"} Jul 19 15:25:21.436139 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:21.436109 2559 generic.go:356] "Generic (PLEG): container finished" podID="9835bbee-48a4-4623-ba54-9cca570c4b6f" containerID="1f53946f50df1ae66b7380983cbc2c454589baa29ab7e4cabeef599aabce5f51" exitCode=0 Jul 19 15:25:21.436482 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:21.436185 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-140663705906320" event={"ID":"9835bbee-48a4-4623-ba54-9cca570c4b6f","Type":"ContainerDied","Data":"1f53946f50df1ae66b7380983cbc2c454589baa29ab7e4cabeef599aabce5f51"} Jul 19 15:25:22.566410 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:22.566384 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705906320" Jul 19 15:25:22.588627 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:22.588599 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7292-140663705906320_9835bbee-48a4-4623-ba54-9cca570c4b6f/test-curl-7292-140663705906320/0.log" Jul 19 15:25:22.612669 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:22.612627 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7292-140663705906320"] Jul 19 15:25:22.614649 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:22.614623 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7292-140663705906320"] Jul 19 15:25:22.715676 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:22.715563 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv2l4\" (UniqueName: \"kubernetes.io/projected/9835bbee-48a4-4623-ba54-9cca570c4b6f-kube-api-access-kv2l4\") pod \"9835bbee-48a4-4623-ba54-9cca570c4b6f\" (UID: \"9835bbee-48a4-4623-ba54-9cca570c4b6f\") " Jul 19 15:25:22.717727 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:22.717699 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9835bbee-48a4-4623-ba54-9cca570c4b6f-kube-api-access-kv2l4" (OuterVolumeSpecName: "kube-api-access-kv2l4") pod "9835bbee-48a4-4623-ba54-9cca570c4b6f" (UID: "9835bbee-48a4-4623-ba54-9cca570c4b6f"). InnerVolumeSpecName "kube-api-access-kv2l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:25:22.816815 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:22.816779 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv2l4\" (UniqueName: \"kubernetes.io/projected/9835bbee-48a4-4623-ba54-9cca570c4b6f-kube-api-access-kv2l4\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 19 15:25:23.448254 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:23.448225 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d3282d1f0d3afedef81f77bd925df42a765b15bddbdd7b8fb0abc903e9676e" Jul 19 15:25:23.448427 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:23.448260 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-140663705906320" Jul 19 15:25:23.457929 ip-10-0-134-156 kubenswrapper[2559]: E0719 15:25:23.457901 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7292-140663705906320\" is forbidden: User \"system:node:ip-10-0-134-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-156.ec2.internal' and this object" podUID="9835bbee-48a4-4623-ba54-9cca570c4b6f" pod="opendatahub/test-curl-7292-140663705906320" Jul 19 15:25:24.107062 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:24.107028 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9835bbee-48a4-4623-ba54-9cca570c4b6f" path="/var/lib/kubelet/pods/9835bbee-48a4-4623-ba54-9cca570c4b6f/volumes" Jul 19 15:25:36.396786 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:36.396753 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:25:58.493269 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:25:58.493184 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8x9kp"] Jul 19 15:26:07.082352 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:07.082314 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77c6ff4774-ngpxf_fb919240-ff76-4e7a-847c-48a8f7f48777/authorino/0.log" Jul 19 15:26:13.221120 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:13.221090 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56_30b720f1-59a3-4905-9a2c-5bd386c81d12/copy-manifests/0.log" Jul 19 15:26:13.227783 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:13.227756 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56_30b720f1-59a3-4905-9a2c-5bd386c81d12/manager/0.log" Jul 19 15:26:14.828883 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:14.828835 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7_8949ef80-3fe1-4348-aebf-917365646e05/util/0.log" Jul 19 15:26:14.835276 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:14.835258 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7_8949ef80-3fe1-4348-aebf-917365646e05/pull/0.log" Jul 19 15:26:14.841420 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:14.841403 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7_8949ef80-3fe1-4348-aebf-917365646e05/extract/0.log" Jul 19 15:26:14.950377 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:14.950355 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs_d1b5f999-9e62-40a4-86d8-b8d2b745aa11/util/0.log" Jul 19 15:26:14.956351 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:14.956332 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs_d1b5f999-9e62-40a4-86d8-b8d2b745aa11/pull/0.log" Jul 19 15:26:14.964653 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:14.964635 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs_d1b5f999-9e62-40a4-86d8-b8d2b745aa11/extract/0.log" Jul 19 15:26:15.083567 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:15.083479 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8_3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842/util/0.log" Jul 19 15:26:15.089961 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:15.089938 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8_3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842/pull/0.log" Jul 19 15:26:15.097043 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:15.097024 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8_3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842/extract/0.log" Jul 19 15:26:15.211626 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:15.211598 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk_b90b0b29-94c7-4369-933f-b88985ca6de9/util/0.log" Jul 19 15:26:15.217937 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:15.217917 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk_b90b0b29-94c7-4369-933f-b88985ca6de9/pull/0.log" Jul 19 15:26:15.223930 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:15.223912 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk_b90b0b29-94c7-4369-933f-b88985ca6de9/extract/0.log" Jul 19 15:26:15.344737 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:15.344648 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77c6ff4774-ngpxf_fb919240-ff76-4e7a-847c-48a8f7f48777/authorino/0.log" Jul 19 15:26:15.702760 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:15.702677 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jpm62_509523fe-2db9-4213-9c7d-cbbf23101843/kuadrant-console-plugin/0.log" Jul 19 15:26:16.069907 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:16.069814 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8x9kp_c1f07116-eaec-412e-8e8f-bb9c806a03eb/limitador/0.log" Jul 19 15:26:16.900164 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:16.900134 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fc68cb667-vt9qv_2e021d53-1b40-42a6-8889-2fc1c5674287/kube-auth-proxy/0.log" Jul 19 15:26:17.397169 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:17.397138 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66bd9878b6-whfdj_05139392-8ed3-48d4-91b1-f6a56d0b4e5d/router/0.log" Jul 19 15:26:17.973743 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:17.973705 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-z6rjk_cf39c267-14d5-4bfd-ad66-a022327fd493/main/0.log" Jul 19 15:26:18.316691 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:18.316616 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-wt5dk_78f3c579-56f0-4e60-9d81-e1ddaab26c08/main/0.log" Jul 19 15:26:24.163426 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.163388 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9q7mm/must-gather-qhcsp"] Jul 19 15:26:24.163800 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.163732 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9835bbee-48a4-4623-ba54-9cca570c4b6f" containerName="test-curl-7292-140663705906320" Jul 19 15:26:24.163800 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.163743 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9835bbee-48a4-4623-ba54-9cca570c4b6f" containerName="test-curl-7292-140663705906320" Jul 19 15:26:24.163909 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.163809 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9835bbee-48a4-4623-ba54-9cca570c4b6f" containerName="test-curl-7292-140663705906320" Jul 19 15:26:24.167051 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.167034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9q7mm/must-gather-qhcsp" Jul 19 15:26:24.169340 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.169324 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9q7mm\"/\"kube-root-ca.crt\"" Jul 19 15:26:24.170388 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.170360 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9q7mm\"/\"openshift-service-ca.crt\"" Jul 19 15:26:24.170530 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.170361 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9q7mm\"/\"default-dockercfg-8hr7t\"" Jul 19 15:26:24.176257 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.176233 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9q7mm/must-gather-qhcsp"] Jul 19 15:26:24.251226 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.251191 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczqb\" (UniqueName: \"kubernetes.io/projected/5c5d90b9-6d88-44be-b223-a1fbb4840036-kube-api-access-jczqb\") pod \"must-gather-qhcsp\" (UID: \"5c5d90b9-6d88-44be-b223-a1fbb4840036\") " pod="openshift-must-gather-9q7mm/must-gather-qhcsp" Jul 19 15:26:24.251397 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.251239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5c5d90b9-6d88-44be-b223-a1fbb4840036-must-gather-output\") pod \"must-gather-qhcsp\" (UID: \"5c5d90b9-6d88-44be-b223-a1fbb4840036\") " pod="openshift-must-gather-9q7mm/must-gather-qhcsp" Jul 19 15:26:24.352200 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.352164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5c5d90b9-6d88-44be-b223-a1fbb4840036-must-gather-output\") pod \"must-gather-qhcsp\" (UID: \"5c5d90b9-6d88-44be-b223-a1fbb4840036\") " pod="openshift-must-gather-9q7mm/must-gather-qhcsp" Jul 19 15:26:24.352395 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.352254 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczqb\" (UniqueName: \"kubernetes.io/projected/5c5d90b9-6d88-44be-b223-a1fbb4840036-kube-api-access-jczqb\") pod \"must-gather-qhcsp\" (UID: \"5c5d90b9-6d88-44be-b223-a1fbb4840036\") " pod="openshift-must-gather-9q7mm/must-gather-qhcsp" Jul 19 15:26:24.352560 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.352535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5c5d90b9-6d88-44be-b223-a1fbb4840036-must-gather-output\") pod \"must-gather-qhcsp\" (UID: \"5c5d90b9-6d88-44be-b223-a1fbb4840036\") " pod="openshift-must-gather-9q7mm/must-gather-qhcsp" Jul 19 15:26:24.359739 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.359709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczqb\" (UniqueName: \"kubernetes.io/projected/5c5d90b9-6d88-44be-b223-a1fbb4840036-kube-api-access-jczqb\") pod \"must-gather-qhcsp\" (UID: \"5c5d90b9-6d88-44be-b223-a1fbb4840036\") " pod="openshift-must-gather-9q7mm/must-gather-qhcsp" Jul 19 15:26:24.477283 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.477184 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9q7mm/must-gather-qhcsp" Jul 19 15:26:24.611811 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.611779 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9q7mm/must-gather-qhcsp"] Jul 19 15:26:24.613215 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:26:24.613190 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c5d90b9_6d88_44be_b223_a1fbb4840036.slice/crio-f1838d025c1bd61f2ed536832a33adaef3ba6acb7f1d27a8dc3a2c93c8b035e5 WatchSource:0}: Error finding container f1838d025c1bd61f2ed536832a33adaef3ba6acb7f1d27a8dc3a2c93c8b035e5: Status 404 returned error can't find the container with id f1838d025c1bd61f2ed536832a33adaef3ba6acb7f1d27a8dc3a2c93c8b035e5 Jul 19 15:26:24.770756 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:24.770668 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9q7mm/must-gather-qhcsp" event={"ID":"5c5d90b9-6d88-44be-b223-a1fbb4840036","Type":"ContainerStarted","Data":"f1838d025c1bd61f2ed536832a33adaef3ba6acb7f1d27a8dc3a2c93c8b035e5"} Jul 19 15:26:25.780384 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:25.780289 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9q7mm/must-gather-qhcsp" event={"ID":"5c5d90b9-6d88-44be-b223-a1fbb4840036","Type":"ContainerStarted","Data":"0c9733bf4a7ce12697bed644053cdf652415e1c25ee4244405d3abc0fa1c547a"} Jul 19 15:26:25.780384 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:25.780341 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9q7mm/must-gather-qhcsp" event={"ID":"5c5d90b9-6d88-44be-b223-a1fbb4840036","Type":"ContainerStarted","Data":"ad91bc9cc3c7264da1c7a403d91c2d03905863283b31def9f021d3ca62c36095"} Jul 19 15:26:25.795999 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:25.795937 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9q7mm/must-gather-qhcsp" podStartSLOduration=0.991893297 podStartE2EDuration="1.795914372s" totalImagesPullingTime="804.021075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:26:25.794317644 +0000 UTC m=+2998.216541976" watchObservedRunningTime="2026-07-19 15:26:25.795914372 +0000 UTC m=+2998.218138704" Jul 19 15:26:26.927324 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:26.927294 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ffhs6_522a5dd5-0b75-4721-830e-42417948a408/global-pull-secret-syncer/0.log" Jul 19 15:26:27.053953 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:27.053920 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cdbwf_300ecf05-298e-4db7-b26d-f7b7abeb1855/konnectivity-agent/0.log" Jul 19 15:26:27.106035 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:27.106008 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-156.ec2.internal_3d3debe7f1b24cd0f146d976ae1a3f64/haproxy/0.log" Jul 19 15:26:28.369996 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:28.369963 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:26:28.380798 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:28.380763 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:26:31.329095 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.329067 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7_8949ef80-3fe1-4348-aebf-917365646e05/extract/0.log" Jul 19 15:26:31.353016 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.352984 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7_8949ef80-3fe1-4348-aebf-917365646e05/util/0.log" Jul 19 15:26:31.374717 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.374682 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599bbc7_8949ef80-3fe1-4348-aebf-917365646e05/pull/0.log" Jul 19 15:26:31.404494 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.404464 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs_d1b5f999-9e62-40a4-86d8-b8d2b745aa11/extract/0.log" Jul 19 15:26:31.431411 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.431386 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs_d1b5f999-9e62-40a4-86d8-b8d2b745aa11/util/0.log" Jul 19 15:26:31.455563 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.455531 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q8sqs_d1b5f999-9e62-40a4-86d8-b8d2b745aa11/pull/0.log" Jul 19 15:26:31.494321 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.494291 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8_3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842/extract/0.log" Jul 19 15:26:31.515902 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.515807 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8_3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842/util/0.log" Jul 19 15:26:31.537266 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.537237 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bpvb8_3ac0d3b7-36d7-4d3d-bc4c-863bebdb2842/pull/0.log" Jul 19 15:26:31.563926 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.563859 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk_b90b0b29-94c7-4369-933f-b88985ca6de9/extract/0.log" Jul 19 15:26:31.590121 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.590076 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk_b90b0b29-94c7-4369-933f-b88985ca6de9/util/0.log" Jul 19 15:26:31.616056 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:31.616026 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pd4gk_b90b0b29-94c7-4369-933f-b88985ca6de9/pull/0.log" Jul 19 15:26:32.072713 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:32.072678 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77c6ff4774-ngpxf_fb919240-ff76-4e7a-847c-48a8f7f48777/authorino/0.log" Jul 19 15:26:32.201590 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:32.201558 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jpm62_509523fe-2db9-4213-9c7d-cbbf23101843/kuadrant-console-plugin/0.log" Jul 19 15:26:32.500340 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:32.500245 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8x9kp_c1f07116-eaec-412e-8e8f-bb9c806a03eb/limitador/0.log" Jul 19 15:26:34.023733 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.023684 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6ccf501-4751-47b9-9573-1e6cdf51748c/alertmanager/0.log" Jul 19 15:26:34.045382 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.045342 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6ccf501-4751-47b9-9573-1e6cdf51748c/config-reloader/0.log" Jul 19 15:26:34.069563 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.069531 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6ccf501-4751-47b9-9573-1e6cdf51748c/kube-rbac-proxy-web/0.log" Jul 19 15:26:34.089676 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.089641 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6ccf501-4751-47b9-9573-1e6cdf51748c/kube-rbac-proxy/0.log" Jul 19 15:26:34.139563 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.139535 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6ccf501-4751-47b9-9573-1e6cdf51748c/kube-rbac-proxy-metric/0.log" Jul 19 15:26:34.175695 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.175662 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6ccf501-4751-47b9-9573-1e6cdf51748c/prom-label-proxy/0.log" Jul 19 15:26:34.201726 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.201565 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6ccf501-4751-47b9-9573-1e6cdf51748c/init-config-reloader/0.log" Jul 19 15:26:34.252213 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.252180 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-8gkjf_b8785e97-0280-4335-a04c-823904ee8e2f/cluster-monitoring-operator/0.log" Jul 19 15:26:34.362196 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.362155 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76f4d76857-x8krn_b6fb01f4-f68c-4d47-95d0-89728ace8e1c/metrics-server/0.log" Jul 19 15:26:34.509890 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.509830 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75spp_ebbcac45-6d88-4eaf-83f9-e1520f88b049/node-exporter/0.log" Jul 19 15:26:34.531704 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.531669 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75spp_ebbcac45-6d88-4eaf-83f9-e1520f88b049/kube-rbac-proxy/0.log" Jul 19 15:26:34.566691 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.566653 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75spp_ebbcac45-6d88-4eaf-83f9-e1520f88b049/init-textfile/0.log" Jul 19 15:26:34.912253 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.912188 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bpqtk_c4fc61cf-4d7b-4764-84b3-f05e0dbe2402/prometheus-operator/0.log" Jul 19 15:26:34.930755 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:34.930719 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bpqtk_c4fc61cf-4d7b-4764-84b3-f05e0dbe2402/kube-rbac-proxy/0.log" Jul 19 15:26:36.363298 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:36.363271 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-2dcp2_30c83ee5-7890-4240-957c-49de880b15a4/networking-console-plugin/0.log" Jul 19 15:26:36.969651 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:36.969613 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt"] Jul 19 15:26:36.975045 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:36.975017 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:36.980422 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:36.980394 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt"] Jul 19 15:26:37.005718 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.003250 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/1.log" Jul 19 15:26:37.008706 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.008620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-sys\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.008706 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.008672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-podres\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.008889 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.008789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-proc\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.008889 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.008818 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-lib-modules\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.008990 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.008942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pqh9\" (UniqueName: \"kubernetes.io/projected/14b8d69d-0e7c-47b7-8430-0f443d341dc3-kube-api-access-9pqh9\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.009931 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.009907 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-74gwb_69e1fb02-21d0-4cb1-b7fc-f5fb1ba93ac8/console-operator/2.log" Jul 19 15:26:37.109974 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.109933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-sys\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.110202 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.109984 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-podres\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.110202 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.110040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-proc\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.110202 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.110066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-lib-modules\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.110202 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.110109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqh9\" (UniqueName: \"kubernetes.io/projected/14b8d69d-0e7c-47b7-8430-0f443d341dc3-kube-api-access-9pqh9\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.110607 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.110581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-sys\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.110704 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.110661 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-proc\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.110801 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.110784 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-podres\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.110849 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.110840 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14b8d69d-0e7c-47b7-8430-0f443d341dc3-lib-modules\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.119248 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.119213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pqh9\" (UniqueName: \"kubernetes.io/projected/14b8d69d-0e7c-47b7-8430-0f443d341dc3-kube-api-access-9pqh9\") pod \"perf-node-gather-daemonset-g7cdt\" (UID: \"14b8d69d-0e7c-47b7-8430-0f443d341dc3\") " pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.289312 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.289221 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.480031 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.479995 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt"] Jul 19 15:26:37.489441 ip-10-0-134-156 kubenswrapper[2559]: W0719 15:26:37.489402 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod14b8d69d_0e7c_47b7_8430_0f443d341dc3.slice/crio-706bf669b1dfd693e4825be7304a18ccfaeb6859f3ddf8427caa8cebe2a269ce WatchSource:0}: Error finding container 706bf669b1dfd693e4825be7304a18ccfaeb6859f3ddf8427caa8cebe2a269ce: Status 404 returned error can't find the container with id 706bf669b1dfd693e4825be7304a18ccfaeb6859f3ddf8427caa8cebe2a269ce Jul 19 15:26:37.878637 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.878598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" event={"ID":"14b8d69d-0e7c-47b7-8430-0f443d341dc3","Type":"ContainerStarted","Data":"f661022b487bc31824e7083d0f36912ad91975cd5522a47f044b77acae6be05f"} Jul 19 15:26:37.878820 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.878664 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" event={"ID":"14b8d69d-0e7c-47b7-8430-0f443d341dc3","Type":"ContainerStarted","Data":"706bf669b1dfd693e4825be7304a18ccfaeb6859f3ddf8427caa8cebe2a269ce"} Jul 19 15:26:37.878820 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.878714 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:37.895852 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:37.895787 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" podStartSLOduration=1.8957655930000001 podStartE2EDuration="1.895765593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 15:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:26:37.892926436 +0000 UTC m=+3010.315150767" watchObservedRunningTime="2026-07-19 15:26:37.895765593 +0000 UTC m=+3010.317989925" Jul 19 15:26:39.452965 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:39.452933 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bgdc6_3664d65b-c2f1-4254-a294-9f4123067a5b/dns/0.log" Jul 19 15:26:39.472810 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:39.472787 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bgdc6_3664d65b-c2f1-4254-a294-9f4123067a5b/kube-rbac-proxy/0.log" Jul 19 15:26:39.516490 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:39.516462 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7rprf_e0307f2d-fa2f-42dd-984e-148f44510d07/dns-node-resolver/0.log" Jul 19 15:26:40.108234 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:40.108196 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lgdc4_7b223bb6-1dec-46f4-a6a8-7204b718d631/node-ca/0.log" Jul 19 15:26:41.296297 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:41.296266 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fc68cb667-vt9qv_2e021d53-1b40-42a6-8889-2fc1c5674287/kube-auth-proxy/0.log" Jul 19 15:26:41.461772 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:41.461739 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66bd9878b6-whfdj_05139392-8ed3-48d4-91b1-f6a56d0b4e5d/router/0.log" Jul 19 15:26:42.128353 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:42.128321 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tkvk_017827df-343d-42db-beec-ce9b21b305f5/serve-healthcheck-canary/0.log" Jul 19 15:26:42.627220 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:42.627185 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-sjhrl_65b96bc2-0209-4d7d-aa1c-a2348b7b14b9/insights-operator/0.log" Jul 19 15:26:42.627703 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:42.627623 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-sjhrl_65b96bc2-0209-4d7d-aa1c-a2348b7b14b9/insights-operator/1.log" Jul 19 15:26:42.787267 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:42.787235 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v69md_6278e4df-2a03-458e-bcac-330d3ab063cc/kube-rbac-proxy/0.log" Jul 19 15:26:42.806847 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:42.806820 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v69md_6278e4df-2a03-458e-bcac-330d3ab063cc/exporter/0.log" Jul 19 15:26:42.828670 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:42.828638 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v69md_6278e4df-2a03-458e-bcac-330d3ab063cc/extractor/0.log" Jul 19 15:26:43.901400 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:43.901370 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9q7mm/perf-node-gather-daemonset-g7cdt" Jul 19 15:26:45.525441 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:45.525413 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56_30b720f1-59a3-4905-9a2c-5bd386c81d12/manager/0.log" Jul 19 15:26:45.544831 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:45.544804 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c4b5b7c5f-fwf56_30b720f1-59a3-4905-9a2c-5bd386c81d12/copy-manifests/0.log" Jul 19 15:26:46.976716 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:46.976678 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fc68cb667-vt9qv_2e021d53-1b40-42a6-8889-2fc1c5674287/kube-auth-proxy/0.log" Jul 19 15:26:47.112179 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:47.112154 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66bd9878b6-whfdj_05139392-8ed3-48d4-91b1-f6a56d0b4e5d/router/0.log" Jul 19 15:26:47.152014 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:47.151939 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7499d794d8-497rt_58079111-f079-4364-8c69-47bff67e612f/manager/0.log" Jul 19 15:26:47.204671 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:47.204604 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-dh65k_c73e9315-c657-478c-a100-7f7c1a5bd5d7/openshift-lws-operator/0.log" Jul 19 15:26:48.414122 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:48.414038 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fc68cb667-vt9qv_2e021d53-1b40-42a6-8889-2fc1c5674287/kube-auth-proxy/0.log" Jul 19 15:26:48.564375 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:48.564338 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66bd9878b6-whfdj_05139392-8ed3-48d4-91b1-f6a56d0b4e5d/router/0.log" Jul 19 15:26:53.500611 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.500584 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qnjcn_096a094e-c379-445d-8e2f-5eb81a93f05a/kube-multus-additional-cni-plugins/0.log" Jul 19 15:26:53.522270 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.522243 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qnjcn_096a094e-c379-445d-8e2f-5eb81a93f05a/egress-router-binary-copy/0.log" Jul 19 15:26:53.543366 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.543339 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qnjcn_096a094e-c379-445d-8e2f-5eb81a93f05a/cni-plugins/0.log" Jul 19 15:26:53.563214 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.563190 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qnjcn_096a094e-c379-445d-8e2f-5eb81a93f05a/bond-cni-plugin/0.log" Jul 19 15:26:53.583595 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.583566 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qnjcn_096a094e-c379-445d-8e2f-5eb81a93f05a/routeoverride-cni/0.log" Jul 19 15:26:53.602635 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.602610 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qnjcn_096a094e-c379-445d-8e2f-5eb81a93f05a/whereabouts-cni-bincopy/0.log" Jul 19 15:26:53.622284 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.622259 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qnjcn_096a094e-c379-445d-8e2f-5eb81a93f05a/whereabouts-cni/0.log" Jul 19 15:26:53.662560 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.662526 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cxvqm_a756c7f8-fc87-4f5b-9d5c-803a4d580084/kube-multus/0.log" Jul 19 15:26:53.834312 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.834262 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zw8ds_82fc003a-678c-4e3d-90ab-0c64f693f817/network-metrics-daemon/0.log" Jul 19 15:26:53.855069 ip-10-0-134-156 kubenswrapper[2559]: I0719 15:26:53.855038 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zw8ds_82fc003a-678c-4e3d-90ab-0c64f693f817/kube-rbac-proxy/0.log"