Jul 21 19:37:16.723198 ip-10-0-134-158 systemd[1]: Starting Kubernetes Kubelet... Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.027385 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031868 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031894 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031897 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031901 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031904 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031908 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031911 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 19:37:17.082096 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031914 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031916 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031919 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031922 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031925 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031929 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031933 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031936 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031939 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031942 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031944 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031947 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031950 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031952 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031955 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031959 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031964 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031967 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031970 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 19:37:17.083927 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031973 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031975 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031978 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031980 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031983 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031985 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031989 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031992 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031994 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031997 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.031999 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032002 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032004 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032007 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032010 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032013 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032016 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032019 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032021 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 19:37:17.084632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032024 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032026 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032029 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032033 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032035 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032038 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032041 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032045 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032047 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032050 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032053 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032055 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032058 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032061 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032064 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032067 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032069 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032072 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032074 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032077 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 19:37:17.086255 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032080 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032082 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032085 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032087 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032090 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032094 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032097 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032100 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032102 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032105 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032107 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032109 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032112 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032115 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032117 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032121 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032125 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032128 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032131 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032133 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 19:37:17.087602 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032136 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032139 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032143 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032145 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032148 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032151 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032154 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032156 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032159 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032161 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032164 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032166 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032169 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.032171 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034709 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034721 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034726 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034729 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034732 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034735 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 19:37:17.116850 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034738 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 19:37:17.089357 ip-10-0-134-158 systemd[1]: Started Kubernetes Kubelet. Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034741 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034744 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034747 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034749 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034752 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034754 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034757 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034759 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034762 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034764 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034767 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034769 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034772 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034774 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034777 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034779 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034782 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034785 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034787 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 19:37:17.120868 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034790 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034794 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034798 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034801 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034805 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034808 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034811 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034814 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034817 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034820 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034822 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034825 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034827 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034830 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034832 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034835 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034838 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034840 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034843 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 19:37:17.124139 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034845 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034848 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034850 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034853 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034855 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034858 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034860 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034863 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034866 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034868 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034889 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034892 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034896 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034901 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034904 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034907 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034909 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034912 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034915 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 19:37:17.128079 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034918 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034921 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034924 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034927 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034929 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034932 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034934 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034938 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034941 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034943 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034947 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034951 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034953 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034956 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034958 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034961 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034963 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034966 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034968 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034972 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 19:37:17.130705 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034975 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034977 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034980 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034983 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034986 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034989 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034991 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034994 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.034997 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035000 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035003 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035006 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035008 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035011 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035013 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035410 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035415 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035419 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035422 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 19:37:17.134375 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035425 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035428 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035431 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035434 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035436 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035439 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035442 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035445 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035447 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035450 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035453 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035455 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035458 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035461 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035464 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035466 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035469 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035473 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035476 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035478 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 19:37:17.137091 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035481 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035484 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035486 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035489 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035491 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035494 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035496 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035499 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035502 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035504 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035507 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035510 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035513 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035515 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035518 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035521 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035523 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035526 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035528 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035533 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 19:37:17.140571 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035535 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035538 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035540 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035543 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035546 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035548 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035551 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035553 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035556 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035559 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035561 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035564 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035566 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035569 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035571 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035574 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035576 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035579 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035581 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035585 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 19:37:17.143487 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035588 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035591 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035594 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035598 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035601 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035603 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035606 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035608 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035611 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035614 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035618 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035623 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035626 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035629 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035632 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035634 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035637 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035641 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035647 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 19:37:17.144280 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035650 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035652 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035655 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035657 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035660 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035662 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035665 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035668 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035670 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035672 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035675 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035678 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035680 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035682 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035685 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.035687 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035770 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035778 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035783 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035788 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035791 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 19:37:17.145235 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035795 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035799 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035803 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035808 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035811 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035814 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035818 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035821 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035824 2557 flags.go:64] FLAG: --cgroup-root="" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035827 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035831 2557 flags.go:64] FLAG: --client-ca-file="" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035835 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035838 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035842 2557 flags.go:64] FLAG: --cluster-domain="" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035845 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035848 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035851 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035855 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035858 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035861 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035864 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035867 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035887 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035891 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 19:37:17.146619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035894 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035897 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035900 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035905 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035908 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035910 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035913 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035916 2557 flags.go:64] FLAG: --enable-server="true" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035919 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035923 2557 flags.go:64] FLAG: --event-burst="100" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035926 2557 flags.go:64] FLAG: --event-qps="50" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035931 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035933 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035936 2557 flags.go:64] FLAG: --eviction-hard="" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035940 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035943 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035946 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035949 2557 flags.go:64] FLAG: --eviction-soft="" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035953 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035956 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035959 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035962 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035965 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035968 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035971 2557 flags.go:64] FLAG: --feature-gates="" Jul 21 19:37:17.147940 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035974 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035977 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035980 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035983 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035986 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035989 2557 flags.go:64] FLAG: --help="false" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035992 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-134-158.ec2.internal" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035995 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.035998 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036001 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036004 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036008 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036011 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036013 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036016 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036019 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036022 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036025 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036029 2557 flags.go:64] FLAG: --kube-reserved="" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036032 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036035 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036038 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036040 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036043 2557 flags.go:64] FLAG: --lock-file="" Jul 21 19:37:17.149284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036046 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036050 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036053 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036059 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036063 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036065 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036068 2557 flags.go:64] FLAG: --logging-format="text" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036071 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036074 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036077 2557 flags.go:64] FLAG: --manifest-url="" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036080 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036084 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036087 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036091 2557 flags.go:64] FLAG: --max-pods="110" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036094 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036097 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036099 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036102 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036105 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036108 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036111 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036118 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036121 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036124 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 19:37:17.155679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036127 2557 flags.go:64] FLAG: --pod-cidr="" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036130 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036135 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036139 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036142 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036145 2557 flags.go:64] FLAG: --port="10250" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036148 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036151 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09488622d033f0686" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036155 2557 flags.go:64] FLAG: --qos-reserved="" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036160 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036163 2557 flags.go:64] FLAG: --register-node="true" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036166 2557 flags.go:64] FLAG: --register-with-taints="" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036169 2557 flags.go:64] FLAG: --registry-burst="10" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036177 2557 flags.go:64] FLAG: --registry-qps="5" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036180 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036183 2557 flags.go:64] FLAG: --reserved-memory="" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036187 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036189 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036192 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036195 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036198 2557 flags.go:64] FLAG: --runonce="false" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036201 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036204 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036207 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036210 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 19:37:17.157046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036212 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036215 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036218 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036221 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036224 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036227 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036230 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036232 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036235 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036238 2557 flags.go:64] FLAG: --system-cgroups="" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036242 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036247 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036250 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036253 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036257 2557 flags.go:64] FLAG: --tls-min-version="" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036259 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036264 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036267 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036270 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036273 2557 flags.go:64] FLAG: --v="2" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036277 2557 flags.go:64] FLAG: --version="false" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036282 2557 flags.go:64] FLAG: --vmodule="" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036285 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.036288 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036423 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 19:37:17.167802 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036427 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036430 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036433 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036436 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036438 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036441 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036444 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036447 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036449 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036452 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036455 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036457 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036459 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036462 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036465 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036467 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036470 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036474 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036477 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036479 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 19:37:17.169164 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036482 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036484 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036487 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036490 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036493 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036496 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036498 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036501 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036503 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036506 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036511 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036514 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036517 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036520 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036522 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036525 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036528 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036530 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036532 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036535 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 19:37:17.170094 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036538 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036541 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036543 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036546 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036548 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036551 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036553 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036556 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036558 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036562 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036564 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036567 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036569 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036572 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036575 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036578 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036581 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036584 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036586 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036589 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 19:37:17.171003 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036591 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036594 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036597 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036600 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036603 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036605 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036608 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036610 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036613 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036615 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036618 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036620 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036623 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036625 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036627 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036630 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036633 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036635 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036637 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036640 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 19:37:17.171918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036643 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036646 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036649 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036652 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036654 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036657 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036661 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036666 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036670 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036674 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036676 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036679 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036682 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036684 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036687 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036690 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036693 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 19:37:17.260235 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.036696 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.037187 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.043464 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.043481 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.043500 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.043506 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043560 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043566 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043569 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043572 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043576 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043579 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043582 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043585 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043588 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 19:37:17.283219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043591 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043593 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043596 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043599 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043602 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043604 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043607 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043610 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043612 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043615 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043617 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043620 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043622 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043625 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043628 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043632 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043635 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043638 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043641 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 19:37:17.283960 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043645 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043649 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043652 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043655 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043658 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043660 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043663 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043665 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043668 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043671 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043673 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043676 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043679 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043682 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043684 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043687 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043691 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043694 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043697 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043700 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 19:37:17.284632 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043703 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043706 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043708 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043711 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043714 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043716 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043719 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043721 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043724 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043726 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043729 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043732 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043734 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043737 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043740 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043743 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043746 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043748 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043751 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043761 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 19:37:17.285331 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043764 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043766 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043769 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043771 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043774 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043777 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043779 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043782 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043785 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043791 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043793 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043796 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043800 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043804 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043808 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043811 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043814 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043816 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043819 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 19:37:17.286050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043821 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043823 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043826 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043829 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043832 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043835 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043837 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043840 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043844 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043846 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043849 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043851 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.043856 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043987 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043992 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 19:37:17.286739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043995 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.043999 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044003 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044006 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044009 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044012 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044014 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044019 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044021 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044024 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044026 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044030 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044033 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044035 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044038 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044041 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044043 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044046 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044049 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 19:37:17.287435 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044051 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044054 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044056 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044059 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044062 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044064 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044067 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044070 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044073 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044075 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044078 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044080 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044083 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044085 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044088 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044090 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044093 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044095 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044098 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044100 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 19:37:17.288179 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044104 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044106 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044109 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044112 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044116 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044118 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044121 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044123 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044126 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044128 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044131 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044134 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044136 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044139 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044141 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044144 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044146 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044149 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044151 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044154 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 19:37:17.288918 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044157 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044160 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044162 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044165 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044168 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044171 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044174 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044177 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044180 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044183 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044186 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044189 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044192 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044195 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044197 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044200 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044204 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044207 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044209 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 19:37:17.289615 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044212 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044214 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044217 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044220 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044224 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044226 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044229 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044231 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044234 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044237 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044239 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044242 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044244 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044247 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044250 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044252 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044255 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044257 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 19:37:17.290313 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:17.044259 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.044264 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.044369 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.046268 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.047360 2557 server.go:1019] "Starting client certificate rotation" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.047455 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.047490 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.050077 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.052315 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.060828 2557 log.go:25] "Validated CRI v1 runtime API" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.062675 2557 log.go:25] "Validated CRI v1 image API" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.063502 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 19:37:17.291028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.065423 2557 fs.go:135] Filesystem UUIDs: map[0b0bd38b-b46d-4193-a87a-ee9c9f3cba24:/dev/nvme0n1p4 5cc223c0-7c3b-4674-9a24-af0b327876c2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 21 19:37:17.291723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.065437 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 19:37:17.292102 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.070133 2557 manager.go:216] Machine: {Timestamp:2026-07-21 19:37:17.069091661 +0000 UTC m=+0.253533353 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101103 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec213af8ea8d0046b7fd61239b88c05a SystemUUID:ec213af8-ea8d-0046-b7fd-61239b88c05a BootID:4f456636-11b3-4de8-a1f0-f862089acfd5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b8:0a:de:8a:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b8:0a:de:8a:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:e7:76:1f:93:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 19:37:17.292102 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.070235 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 19:37:17.292102 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.070311 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 19:37:17.292102 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.072656 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 19:37:17.292102 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.072685 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-158.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 19:37:17.292102 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.073235 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 19:37:17.292102 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.073247 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 19:37:17.292102 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.073255 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.073816 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.074448 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.074891 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.078620 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.078765 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.078792 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.078802 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.078816 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.078825 2557 kubelet.go:398] "Adding apiserver pod source" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.078834 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.079186 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.079201 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.080837 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.082861 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.082930 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084064 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084085 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084095 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084104 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084116 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084125 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084138 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084147 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084156 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084168 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084189 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084197 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084815 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.084828 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.088592 2557 server.go:1268] "Started kubelet" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.088655 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.088849 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.088947 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-158.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.088964 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.089001 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.089061 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.093353 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 19:37:17.295613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.097086 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mgmbr" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.097439 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.097828 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.100682 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.100693 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.100865 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.101233 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.101379 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.101393 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.101490 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.101498 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.101745 2557 factory.go:55] Registering systemd factory Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.101784 2557 factory.go:223] Registration of the systemd container factory successfully Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.102047 2557 factory.go:153] Registering CRI-O factory Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.102061 2557 factory.go:223] Registration of the crio container factory successfully Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.102112 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.102136 2557 factory.go:103] Registering Raw factory Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.102153 2557 manager.go:1196] Started watching for new ooms in manager Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.102591 2557 manager.go:349] Starting recovery of all containers Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.105695 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mgmbr" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.110689 2557 manager.go:354] Recovery completed Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.114422 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.116189 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.116219 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.116232 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.116245 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.116260 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.116761 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.116771 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.116785 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.116762 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.116217 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-158.ec2.internal.18c465367f902034 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-158.ec2.internal,UID:ip-10-0-134-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-158.ec2.internal,},FirstTimestamp:2026-07-21 19:37:17.08835026 +0000 UTC m=+0.272791975,LastTimestamp:2026-07-21 19:37:17.08835026 +0000 UTC m=+0.272791975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-158.ec2.internal,}" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.117587 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 19:37:17.296683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.117609 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.117619 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.118865 2557 policy_none.go:49] "None policy: Start" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.118891 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.118902 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.120939 2557 policy_none.go:47] "Start" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.266693 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.275596 2557 manager.go:337] "Starting Device Plugin manager" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.275626 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.275636 2557 server.go:87] "Starting device plugin registration server" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.275942 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.275957 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.276040 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.276123 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.276132 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.276690 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.276734 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.321187 2557 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-158.ec2.internal\" not found" node="ip-10-0-134-158.ec2.internal" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.376429 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.377476 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.377507 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.377519 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 19:37:17.391613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.377545 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-158.ec2.internal" Jul 21 19:37:17.498507 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:17.425293 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-158.ec2.internal" Jul 21 19:37:17.498507 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.425320 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-158.ec2.internal\": node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:17.498507 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.449456 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:17.597067 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.550493 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:17.710075 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.651316 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:17.822278 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.752061 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:17.931049 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.852671 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:18.047658 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:17.953689 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:18.047658 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.047566 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 19:37:18.055926 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:18.053979 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:18.168550 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.110671 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 19:32:18 +0000 UTC" deadline="2028-01-08 16:23:51.509958724 +0000 UTC" Jul 21 19:37:18.168550 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.110710 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12860h46m33.399252327s" Jul 21 19:37:18.168550 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:18.154409 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:18.281261 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:18.255065 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:18.403063 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:18.356003 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:18.509953 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:18.456630 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:18.509953 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.497138 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 19:37:18.509953 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.498474 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 19:37:18.509953 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.498525 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 19:37:18.509953 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.498555 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 19:37:18.509953 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:18.498599 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 19:37:18.509953 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.501232 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:18.557233 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-158.ec2.internal\" not found" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.598964 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal"] Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.599042 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.600106 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.600139 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.600157 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.602681 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.602814 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.602847 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.603476 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.603509 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.603524 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.603564 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.603590 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.603606 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.605934 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.605983 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.606659 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.606689 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 19:37:18.610596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.606706 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 19:37:18.624907 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.624891 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 19:37:18.676248 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.676222 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/008bd4e4f63a8822603b78b98862ee55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal\" (UID: \"008bd4e4f63a8822603b78b98862ee55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.676383 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.676259 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/008bd4e4f63a8822603b78b98862ee55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal\" (UID: \"008bd4e4f63a8822603b78b98862ee55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.698051 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.698032 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.700891 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.700855 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.702867 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.702854 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.710060 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.710045 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 19:37:18.713014 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.713001 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 19:37:18.713747 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.713732 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 19:37:18.713821 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:18.713797 2557 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.713821 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.713812 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.723824 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.723807 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 19:37:18.723871 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:18.723846 2557 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-158.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.776410 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.776388 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/008bd4e4f63a8822603b78b98862ee55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal\" (UID: \"008bd4e4f63a8822603b78b98862ee55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.776518 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.776488 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/008bd4e4f63a8822603b78b98862ee55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal\" (UID: \"008bd4e4f63a8822603b78b98862ee55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.776518 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.776451 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/008bd4e4f63a8822603b78b98862ee55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal\" (UID: \"008bd4e4f63a8822603b78b98862ee55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.776585 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.776517 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfae3c7bef61f9e9cdfc141005768ce6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-158.ec2.internal\" (UID: \"bfae3c7bef61f9e9cdfc141005768ce6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.776585 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.776546 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/008bd4e4f63a8822603b78b98862ee55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal\" (UID: \"008bd4e4f63a8822603b78b98862ee55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.876842 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.876762 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfae3c7bef61f9e9cdfc141005768ce6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-158.ec2.internal\" (UID: \"bfae3c7bef61f9e9cdfc141005768ce6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" Jul 21 19:37:18.876842 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:18.876816 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfae3c7bef61f9e9cdfc141005768ce6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-158.ec2.internal\" (UID: \"bfae3c7bef61f9e9cdfc141005768ce6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" Jul 21 19:37:19.014303 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.014274 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" Jul 21 19:37:19.016509 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.016488 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" Jul 21 19:37:19.050637 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.050616 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 19:37:19.053982 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.053965 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 19:37:19.079617 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.079601 2557 apiserver.go:52] "Watching apiserver" Jul 21 19:37:19.087566 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.087543 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 19:37:19.089652 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.089625 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-djwk4","openshift-image-registry/node-ca-64pkq","openshift-multus/multus-x26r8","openshift-network-diagnostics/network-check-target-ll8xj","openshift-network-operator/iptables-alerter-k99vr","openshift-ovn-kubernetes/ovnkube-node-bn8h5","kube-system/global-pull-secret-syncer-kd25k","kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal","openshift-multus/multus-additional-cni-plugins-8lbfw","openshift-multus/network-metrics-daemon-fjfkl","kube-system/konnectivity-agent-jqq6h","openshift-cluster-node-tuning-operator/tuned-m4kjl"] Jul 21 19:37:19.092305 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.092285 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.092374 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.092355 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:19.094487 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.094466 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:19.096786 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.096762 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.097207 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.097185 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f46tk\"" Jul 21 19:37:19.097207 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.097202 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 19:37:19.097386 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.097310 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 19:37:19.099032 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.099013 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:19.099100 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.099077 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:19.099373 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.099355 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 19:37:19.099438 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.099413 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 19:37:19.099558 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.099545 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-r8xrp\"" Jul 21 19:37:19.099629 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.099615 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 19:37:19.099689 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.099669 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 19:37:19.101157 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.101140 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.101510 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.101492 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 19:37:19.103590 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.103573 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 19:37:19.103708 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.103693 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.103797 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.103782 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 19:37:19.103984 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.103968 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6jrdp\"" Jul 21 19:37:19.104066 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.104037 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 19:37:19.105865 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.105850 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.106268 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.106244 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 19:37:19.106268 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.106263 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 19:37:19.106423 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.106350 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 19:37:19.106423 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.106359 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 19:37:19.106423 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.106395 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 19:37:19.106558 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.106469 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 19:37:19.106666 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.106651 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qdwq8\"" Jul 21 19:37:19.108226 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.108196 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.108341 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.108260 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 19:37:19.108341 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.108259 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 19:37:19.108634 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.108620 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 19:37:19.108740 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.108723 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bv6wr\"" Jul 21 19:37:19.111566 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.110968 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 19:37:19.111566 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.111199 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 19:37:19.111566 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.111224 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-68cj6\"" Jul 21 19:37:19.111566 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.111405 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.113799 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.113782 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.116403 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.114609 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 19:37:19.116403 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.114787 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 19:37:19.116403 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.115644 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 19:37:19.116403 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.115904 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7jcm2\"" Jul 21 19:37:19.116403 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.116191 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 19:37:19.116403 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.116353 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 19:37:19.116895 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.116872 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 19:37:19.116997 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.116959 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9kwnh\"" Jul 21 19:37:19.117646 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.117625 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.120041 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.120023 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 19:37:19.120123 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.120071 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 19:37:19.120123 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.120080 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:19.120223 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.120128 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:19.120223 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.120132 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6hmtr\"" Jul 21 19:37:19.138020 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.137904 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vj2gw" Jul 21 19:37:19.145277 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.145261 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vj2gw" Jul 21 19:37:19.178415 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-socket-dir-parent\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178421 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q89c\" (UniqueName: \"kubernetes.io/projected/265c7b96-c23f-4084-8ff8-fa2789f82cee-kube-api-access-5q89c\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178439 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-kubelet\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178464 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-var-lib-openvswitch\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178487 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-var-lib-cni-multus\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178505 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-run-multus-certs\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178520 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-systemd-units\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178568 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-slash\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178590 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-cni-netd\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178611 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-run-netns\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178635 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b96f04b2-899b-4860-8654-6e0061a4d0dc-agent-certs\") pod \"konnectivity-agent-jqq6h\" (UID: \"b96f04b2-899b-4860-8654-6e0061a4d0dc\") " pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178653 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-system-cni-dir\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178669 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-cnibin\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178687 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-daemon-config\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.178724 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178711 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zcjn\" (UniqueName: \"kubernetes.io/projected/e8cdaab2-481b-43f1-b75d-45e7cd7e7824-kube-api-access-6zcjn\") pod \"iptables-alerter-k99vr\" (UID: \"e8cdaab2-481b-43f1-b75d-45e7cd7e7824\") " pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178746 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-log-socket\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178764 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/462d9c54-6e71-424a-9489-166c81de167a-ovn-node-metrics-cert\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178784 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b96f04b2-899b-4860-8654-6e0061a4d0dc-konnectivity-ca\") pod \"konnectivity-agent-jqq6h\" (UID: \"b96f04b2-899b-4860-8654-6e0061a4d0dc\") " pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178804 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-etc-kubernetes\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178821 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8cdaab2-481b-43f1-b75d-45e7cd7e7824-iptables-alerter-script\") pod \"iptables-alerter-k99vr\" (UID: \"e8cdaab2-481b-43f1-b75d-45e7cd7e7824\") " pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8cdaab2-481b-43f1-b75d-45e7cd7e7824-host-slash\") pod \"iptables-alerter-k99vr\" (UID: \"e8cdaab2-481b-43f1-b75d-45e7cd7e7824\") " pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178852 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-run-openvswitch\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178872 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178893 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-hostroot\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178941 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-conf-dir\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.178979 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-run-systemd\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179002 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-etc-openvswitch\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179020 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-run-k8s-cni-cncf-io\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.179238 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179069 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-var-lib-kubelet\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179105 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179132 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-run-netns\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179156 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179175 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179192 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5bm\" (UniqueName: \"kubernetes.io/projected/462d9c54-6e71-424a-9489-166c81de167a-kube-api-access-4f5bm\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179219 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aba01382-69a9-4443-b1a2-fe15605e698d-kubelet-config\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179241 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-run-ovn\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179256 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/462d9c54-6e71-424a-9489-166c81de167a-env-overrides\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179270 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/462d9c54-6e71-424a-9489-166c81de167a-ovnkube-script-lib\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179308 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-cni-dir\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179346 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-var-lib-cni-bin\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179374 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-node-log\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.179706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179402 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-cni-bin\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.180087 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179416 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/462d9c54-6e71-424a-9489-166c81de167a-ovnkube-config\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.180087 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179443 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aba01382-69a9-4443-b1a2-fe15605e698d-dbus\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.180087 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179459 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-os-release\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.180087 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.179473 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/265c7b96-c23f-4084-8ff8-fa2789f82cee-cni-binary-copy\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.202526 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.202507 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280054 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-run-multus-certs\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280184 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-run-multus-certs\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280212 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-slash\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280256 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-system-cni-dir\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280281 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-slash\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02c2a5d8-4576-45b1-9c2a-ddbd5012322b-hosts-file\") pod \"node-resolver-djwk4\" (UID: \"02c2a5d8-4576-45b1-9c2a-ddbd5012322b\") " pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280323 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b96f04b2-899b-4860-8654-6e0061a4d0dc-agent-certs\") pod \"konnectivity-agent-jqq6h\" (UID: \"b96f04b2-899b-4860-8654-6e0061a4d0dc\") " pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280357 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-system-cni-dir\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280391 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zcjn\" (UniqueName: \"kubernetes.io/projected/e8cdaab2-481b-43f1-b75d-45e7cd7e7824-kube-api-access-6zcjn\") pod \"iptables-alerter-k99vr\" (UID: \"e8cdaab2-481b-43f1-b75d-45e7cd7e7824\") " pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280431 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-log-socket\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280464 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/462d9c54-6e71-424a-9489-166c81de167a-ovn-node-metrics-cert\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280484 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-system-cni-dir\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.284136 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280536 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-registration-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280567 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-sys\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280595 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b96f04b2-899b-4860-8654-6e0061a4d0dc-konnectivity-ca\") pod \"konnectivity-agent-jqq6h\" (UID: \"b96f04b2-899b-4860-8654-6e0061a4d0dc\") " pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280633 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-etc-kubernetes\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280666 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8cdaab2-481b-43f1-b75d-45e7cd7e7824-host-slash\") pod \"iptables-alerter-k99vr\" (UID: \"e8cdaab2-481b-43f1-b75d-45e7cd7e7824\") " pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280698 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-modprobe-d\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280731 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c2a5d8-4576-45b1-9c2a-ddbd5012322b-tmp-dir\") pod \"node-resolver-djwk4\" (UID: \"02c2a5d8-4576-45b1-9c2a-ddbd5012322b\") " pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280761 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-etc-openvswitch\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280791 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-sys-fs\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280830 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280818 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280867 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5bm\" (UniqueName: \"kubernetes.io/projected/462d9c54-6e71-424a-9489-166c81de167a-kube-api-access-4f5bm\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280948 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-etc-kubernetes\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.285040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.280985 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aba01382-69a9-4443-b1a2-fe15605e698d-kubelet-config\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281023 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281060 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e3f4244-32a9-467b-b77c-26efe45e6aac-host\") pod \"node-ca-64pkq\" (UID: \"0e3f4244-32a9-467b-b77c-26efe45e6aac\") " pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281094 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-sysconfig\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281124 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-cnibin\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281154 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66af5356-d22c-46a3-972b-76f5050acfee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281184 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/462d9c54-6e71-424a-9489-166c81de167a-env-overrides\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281218 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/462d9c54-6e71-424a-9489-166c81de167a-ovnkube-script-lib\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281249 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-sysctl-d\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281350 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-etc-openvswitch\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281564 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-run\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281622 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66af5356-d22c-46a3-972b-76f5050acfee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.281713 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281730 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-log-socket\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.285606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281820 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aba01382-69a9-4443-b1a2-fe15605e698d-kubelet-config\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.281821 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret podName:aba01382-69a9-4443-b1a2-fe15605e698d nodeName:}" failed. No retries permitted until 2026-07-21 19:37:19.781773172 +0000 UTC m=+2.966214835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret") pod "global-pull-secret-syncer-kd25k" (UID: "aba01382-69a9-4443-b1a2-fe15605e698d") : object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281854 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q68v\" (UniqueName: \"kubernetes.io/projected/66af5356-d22c-46a3-972b-76f5050acfee-kube-api-access-8q68v\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281891 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-cni-dir\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281927 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-var-lib-cni-bin\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.281973 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-node-log\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282006 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-cni-bin\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282042 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aba01382-69a9-4443-b1a2-fe15605e698d-dbus\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282074 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-var-lib-kubelet\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282110 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qs7h\" (UniqueName: \"kubernetes.io/projected/02c2a5d8-4576-45b1-9c2a-ddbd5012322b-kube-api-access-4qs7h\") pod \"node-resolver-djwk4\" (UID: \"02c2a5d8-4576-45b1-9c2a-ddbd5012322b\") " pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282141 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282175 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/265c7b96-c23f-4084-8ff8-fa2789f82cee-cni-binary-copy\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282217 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q89c\" (UniqueName: \"kubernetes.io/projected/265c7b96-c23f-4084-8ff8-fa2789f82cee-kube-api-access-5q89c\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.286270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282252 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-var-lib-openvswitch\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282288 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7k95\" (UniqueName: \"kubernetes.io/projected/0e3f4244-32a9-467b-b77c-26efe45e6aac-kube-api-access-q7k95\") pod \"node-ca-64pkq\" (UID: \"0e3f4244-32a9-467b-b77c-26efe45e6aac\") " pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282316 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxzcp\" (UniqueName: \"kubernetes.io/projected/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-kube-api-access-sxzcp\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-var-lib-cni-multus\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282388 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-systemd-units\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282419 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b96f04b2-899b-4860-8654-6e0061a4d0dc-konnectivity-ca\") pod \"konnectivity-agent-jqq6h\" (UID: \"b96f04b2-899b-4860-8654-6e0061a4d0dc\") " pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282425 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-cni-netd\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282454 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-kubernetes\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282487 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-run-netns\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282518 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-cnibin\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282548 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-daemon-config\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282590 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8cdaab2-481b-43f1-b75d-45e7cd7e7824-iptables-alerter-script\") pod \"iptables-alerter-k99vr\" (UID: \"e8cdaab2-481b-43f1-b75d-45e7cd7e7824\") " pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-run-openvswitch\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282660 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e3f4244-32a9-467b-b77c-26efe45e6aac-serviceca\") pod \"node-ca-64pkq\" (UID: \"0e3f4244-32a9-467b-b77c-26efe45e6aac\") " pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.286751 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282687 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-tuned\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282718 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-tmp\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282749 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-hostroot\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282780 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-conf-dir\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282812 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-run-systemd\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282833 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-cni-dir\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282847 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-socket-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282885 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66af5356-d22c-46a3-972b-76f5050acfee-cni-binary-copy\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282891 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-var-lib-cni-bin\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.282942 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-run-k8s-cni-cncf-io\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283032 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-var-lib-cni-multus\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283059 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-cni-netd\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283079 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-systemd-units\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283113 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-run-netns\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283119 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-var-lib-kubelet\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283161 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-node-log\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283204 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-var-lib-kubelet\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283417 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-host-run-k8s-cni-cncf-io\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283471 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-cni-bin\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283516 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-var-lib-openvswitch\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283567 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aba01382-69a9-4443-b1a2-fe15605e698d-dbus\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283759 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/265c7b96-c23f-4084-8ff8-fa2789f82cee-cni-binary-copy\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283801 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-run-netns\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283839 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283879 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283913 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-device-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283964 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-host\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.283985 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8cdaab2-481b-43f1-b75d-45e7cd7e7824-host-slash\") pod \"iptables-alerter-k99vr\" (UID: \"e8cdaab2-481b-43f1-b75d-45e7cd7e7824\") " pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284006 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-run-ovn\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.287641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284042 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-lib-modules\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284075 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ltmh\" (UniqueName: \"kubernetes.io/projected/30e72480-263d-4449-87d6-4a30fdbcfcee-kube-api-access-6ltmh\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284083 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-run-systemd\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284110 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/462d9c54-6e71-424a-9489-166c81de167a-ovnkube-config\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284277 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-run-ovn\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284669 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/462d9c54-6e71-424a-9489-166c81de167a-ovnkube-script-lib\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284831 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkg7\" (UniqueName: \"kubernetes.io/projected/461a408f-8393-47e7-96d2-a8748a4121b1-kube-api-access-7dkg7\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284874 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-systemd\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284882 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/462d9c54-6e71-424a-9489-166c81de167a-ovnkube-config\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284903 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-os-release\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.284975 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-run-openvswitch\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285012 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-os-release\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285011 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-run-netns\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285058 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-socket-dir-parent\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.288086 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285065 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/462d9c54-6e71-424a-9489-166c81de167a-ovn-node-metrics-cert\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285092 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/462d9c54-6e71-424a-9489-166c81de167a-env-overrides\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285105 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-os-release\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-socket-dir-parent\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285280 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b96f04b2-899b-4860-8654-6e0061a4d0dc-agent-certs\") pod \"konnectivity-agent-jqq6h\" (UID: \"b96f04b2-899b-4860-8654-6e0061a4d0dc\") " pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285286 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-hostroot\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285380 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-conf-dir\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285513 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/265c7b96-c23f-4084-8ff8-fa2789f82cee-cnibin\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.285978 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-kubelet\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.286076 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.286102 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-sysctl-conf\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.286146 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/265c7b96-c23f-4084-8ff8-fa2789f82cee-multus-daemon-config\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.286000 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.286039 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/462d9c54-6e71-424a-9489-166c81de167a-host-kubelet\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.288664 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.286401 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8cdaab2-481b-43f1-b75d-45e7cd7e7824-iptables-alerter-script\") pod \"iptables-alerter-k99vr\" (UID: \"e8cdaab2-481b-43f1-b75d-45e7cd7e7824\") " pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.292224 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.292195 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zcjn\" (UniqueName: \"kubernetes.io/projected/e8cdaab2-481b-43f1-b75d-45e7cd7e7824-kube-api-access-6zcjn\") pod \"iptables-alerter-k99vr\" (UID: \"e8cdaab2-481b-43f1-b75d-45e7cd7e7824\") " pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.293020 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.293003 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5bm\" (UniqueName: \"kubernetes.io/projected/462d9c54-6e71-424a-9489-166c81de167a-kube-api-access-4f5bm\") pod \"ovnkube-node-bn8h5\" (UID: \"462d9c54-6e71-424a-9489-166c81de167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.293854 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.293839 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 19:37:19.293901 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.293857 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 19:37:19.293901 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.293867 2557 projected.go:196] Error preparing data for projected volume kube-api-access-llfv6 for pod openshift-network-diagnostics/network-check-target-ll8xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:19.294029 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.293921 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6 podName:18203bb5-2bdc-42cb-ad2f-bfce2b2edd39 nodeName:}" failed. No retries permitted until 2026-07-21 19:37:19.793909371 +0000 UTC m=+2.978351037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-llfv6" (UniqueName: "kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6") pod "network-check-target-ll8xj" (UID: "18203bb5-2bdc-42cb-ad2f-bfce2b2edd39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:19.296788 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.296761 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q89c\" (UniqueName: \"kubernetes.io/projected/265c7b96-c23f-4084-8ff8-fa2789f82cee-kube-api-access-5q89c\") pod \"multus-x26r8\" (UID: \"265c7b96-c23f-4084-8ff8-fa2789f82cee\") " pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.386674 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386644 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.386674 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386678 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e3f4244-32a9-467b-b77c-26efe45e6aac-host\") pod \"node-ca-64pkq\" (UID: \"0e3f4244-32a9-467b-b77c-26efe45e6aac\") " pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386698 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-sysconfig\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386714 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-cnibin\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386770 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e3f4244-32a9-467b-b77c-26efe45e6aac-host\") pod \"node-ca-64pkq\" (UID: \"0e3f4244-32a9-467b-b77c-26efe45e6aac\") " pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386781 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-cnibin\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66af5356-d22c-46a3-972b-76f5050acfee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386803 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-sysconfig\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386825 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386836 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-sysctl-d\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386864 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-run\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.386887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386889 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66af5356-d22c-46a3-972b-76f5050acfee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386917 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q68v\" (UniqueName: \"kubernetes.io/projected/66af5356-d22c-46a3-972b-76f5050acfee-kube-api-access-8q68v\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386937 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-run\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386963 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-var-lib-kubelet\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386972 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-sysctl-d\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.386992 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qs7h\" (UniqueName: \"kubernetes.io/projected/02c2a5d8-4576-45b1-9c2a-ddbd5012322b-kube-api-access-4qs7h\") pod \"node-resolver-djwk4\" (UID: \"02c2a5d8-4576-45b1-9c2a-ddbd5012322b\") " pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387019 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387032 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-var-lib-kubelet\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387049 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7k95\" (UniqueName: \"kubernetes.io/projected/0e3f4244-32a9-467b-b77c-26efe45e6aac-kube-api-access-q7k95\") pod \"node-ca-64pkq\" (UID: \"0e3f4244-32a9-467b-b77c-26efe45e6aac\") " pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387076 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxzcp\" (UniqueName: \"kubernetes.io/projected/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-kube-api-access-sxzcp\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387104 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-kubernetes\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387135 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e3f4244-32a9-467b-b77c-26efe45e6aac-serviceca\") pod \"node-ca-64pkq\" (UID: \"0e3f4244-32a9-467b-b77c-26efe45e6aac\") " pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387178 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-tuned\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387204 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-tmp\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.387456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387243 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-socket-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387270 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66af5356-d22c-46a3-972b-76f5050acfee-cni-binary-copy\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387300 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-device-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387324 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-host\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387342 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66af5356-d22c-46a3-972b-76f5050acfee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387353 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-lib-modules\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ltmh\" (UniqueName: \"kubernetes.io/projected/30e72480-263d-4449-87d6-4a30fdbcfcee-kube-api-access-6ltmh\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387412 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkg7\" (UniqueName: \"kubernetes.io/projected/461a408f-8393-47e7-96d2-a8748a4121b1-kube-api-access-7dkg7\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387439 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-systemd\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387457 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-lib-modules\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387466 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-os-release\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.387491 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387496 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387524 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-sysctl-conf\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.388209 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.387549 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs podName:30e72480-263d-4449-87d6-4a30fdbcfcee nodeName:}" failed. No retries permitted until 2026-07-21 19:37:19.887530371 +0000 UTC m=+3.071972052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs") pod "network-metrics-daemon-fjfkl" (UID: "30e72480-263d-4449-87d6-4a30fdbcfcee") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387487 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66af5356-d22c-46a3-972b-76f5050acfee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387598 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-socket-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387623 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-systemd\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387642 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-sysctl-conf\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387679 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-host\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387683 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-device-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387709 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-os-release\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387733 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387785 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-kubernetes\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387822 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387854 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-system-cni-dir\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387880 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02c2a5d8-4576-45b1-9c2a-ddbd5012322b-hosts-file\") pod \"node-resolver-djwk4\" (UID: \"02c2a5d8-4576-45b1-9c2a-ddbd5012322b\") " pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387927 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-registration-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.388723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.387969 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-sys\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388005 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-modprobe-d\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388032 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c2a5d8-4576-45b1-9c2a-ddbd5012322b-tmp-dir\") pod \"node-resolver-djwk4\" (UID: \"02c2a5d8-4576-45b1-9c2a-ddbd5012322b\") " pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388049 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388060 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-sys-fs\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388053 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02c2a5d8-4576-45b1-9c2a-ddbd5012322b-hosts-file\") pod \"node-resolver-djwk4\" (UID: \"02c2a5d8-4576-45b1-9c2a-ddbd5012322b\") " pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388111 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-sys-fs\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66af5356-d22c-46a3-972b-76f5050acfee-system-cni-dir\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388185 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-modprobe-d\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388209 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/461a408f-8393-47e7-96d2-a8748a4121b1-registration-dir\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388259 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-sys\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388417 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66af5356-d22c-46a3-972b-76f5050acfee-cni-binary-copy\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388438 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c2a5d8-4576-45b1-9c2a-ddbd5012322b-tmp-dir\") pod \"node-resolver-djwk4\" (UID: \"02c2a5d8-4576-45b1-9c2a-ddbd5012322b\") " pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.389149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.388465 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e3f4244-32a9-467b-b77c-26efe45e6aac-serviceca\") pod \"node-ca-64pkq\" (UID: \"0e3f4244-32a9-467b-b77c-26efe45e6aac\") " pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.389692 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.389673 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-tmp\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.389772 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.389755 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-etc-tuned\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.397972 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.397927 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ltmh\" (UniqueName: \"kubernetes.io/projected/30e72480-263d-4449-87d6-4a30fdbcfcee-kube-api-access-6ltmh\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:19.398120 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.398102 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qs7h\" (UniqueName: \"kubernetes.io/projected/02c2a5d8-4576-45b1-9c2a-ddbd5012322b-kube-api-access-4qs7h\") pod \"node-resolver-djwk4\" (UID: \"02c2a5d8-4576-45b1-9c2a-ddbd5012322b\") " pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.398255 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.398234 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7k95\" (UniqueName: \"kubernetes.io/projected/0e3f4244-32a9-467b-b77c-26efe45e6aac-kube-api-access-q7k95\") pod \"node-ca-64pkq\" (UID: \"0e3f4244-32a9-467b-b77c-26efe45e6aac\") " pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.398744 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.398724 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxzcp\" (UniqueName: \"kubernetes.io/projected/4d2572ae-c8ef-469f-bec7-82562ab2cd4a-kube-api-access-sxzcp\") pod \"tuned-m4kjl\" (UID: \"4d2572ae-c8ef-469f-bec7-82562ab2cd4a\") " pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.399018 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.399002 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q68v\" (UniqueName: \"kubernetes.io/projected/66af5356-d22c-46a3-972b-76f5050acfee-kube-api-access-8q68v\") pod \"multus-additional-cni-plugins-8lbfw\" (UID: \"66af5356-d22c-46a3-972b-76f5050acfee\") " pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.399058 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.399028 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkg7\" (UniqueName: \"kubernetes.io/projected/461a408f-8393-47e7-96d2-a8748a4121b1-kube-api-access-7dkg7\") pod \"aws-ebs-csi-driver-node-hllkq\" (UID: \"461a408f-8393-47e7-96d2-a8748a4121b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.421193 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.421172 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:19.437936 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.437910 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x26r8" Jul 21 19:37:19.457245 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.457217 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k99vr" Jul 21 19:37:19.475553 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.475521 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:19.481699 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.481674 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" Jul 21 19:37:19.495208 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.495185 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-djwk4" Jul 21 19:37:19.515132 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.515102 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-64pkq" Jul 21 19:37:19.530210 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.530186 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" Jul 21 19:37:19.535438 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.535417 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" Jul 21 19:37:19.703501 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:19.703467 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008bd4e4f63a8822603b78b98862ee55.slice/crio-698b8ddc3b3a384344257e11d1ea5d18dd8c930f1ce0450f3cb9f53b6194fbd5 WatchSource:0}: Error finding container 698b8ddc3b3a384344257e11d1ea5d18dd8c930f1ce0450f3cb9f53b6194fbd5: Status 404 returned error can't find the container with id 698b8ddc3b3a384344257e11d1ea5d18dd8c930f1ce0450f3cb9f53b6194fbd5 Jul 21 19:37:19.703727 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:19.703701 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfae3c7bef61f9e9cdfc141005768ce6.slice/crio-e65fbb09470e31015054090079e9be22faf0a3cff01dbecf7c278d8221b27f1d WatchSource:0}: Error finding container e65fbb09470e31015054090079e9be22faf0a3cff01dbecf7c278d8221b27f1d: Status 404 returned error can't find the container with id e65fbb09470e31015054090079e9be22faf0a3cff01dbecf7c278d8221b27f1d Jul 21 19:37:19.707789 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.707773 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 19:37:19.791898 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.791845 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:19.792054 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.791975 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:19.792054 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.792028 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret podName:aba01382-69a9-4443-b1a2-fe15605e698d nodeName:}" failed. No retries permitted until 2026-07-21 19:37:20.792013445 +0000 UTC m=+3.976455105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret") pod "global-pull-secret-syncer-kd25k" (UID: "aba01382-69a9-4443-b1a2-fe15605e698d") : object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:19.812078 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:19.812049 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cdaab2_481b_43f1_b75d_45e7cd7e7824.slice/crio-ada205e316d0324f7adcf038d5e4e00194eaf4a513690386a9f571df69d5a318 WatchSource:0}: Error finding container ada205e316d0324f7adcf038d5e4e00194eaf4a513690386a9f571df69d5a318: Status 404 returned error can't find the container with id ada205e316d0324f7adcf038d5e4e00194eaf4a513690386a9f571df69d5a318 Jul 21 19:37:19.842116 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:19.842089 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461a408f_8393_47e7_96d2_a8748a4121b1.slice/crio-8969dfde924b28676c6fa8ea92278b4a6aa80edd04a298d3294d4d8ae05c8f05 WatchSource:0}: Error finding container 8969dfde924b28676c6fa8ea92278b4a6aa80edd04a298d3294d4d8ae05c8f05: Status 404 returned error can't find the container with id 8969dfde924b28676c6fa8ea92278b4a6aa80edd04a298d3294d4d8ae05c8f05 Jul 21 19:37:19.875697 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:19.875673 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c2a5d8_4576_45b1_9c2a_ddbd5012322b.slice/crio-f5cbf693c4b1dd78c0ee849bd41c9edc9557ba8e941538aec918ce8552f16607 WatchSource:0}: Error finding container f5cbf693c4b1dd78c0ee849bd41c9edc9557ba8e941538aec918ce8552f16607: Status 404 returned error can't find the container with id f5cbf693c4b1dd78c0ee849bd41c9edc9557ba8e941538aec918ce8552f16607 Jul 21 19:37:19.878532 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:19.878509 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66af5356_d22c_46a3_972b_76f5050acfee.slice/crio-2eb7ea341b59dd429acd497b228237f119800b8e54ab2c10ecb1d4355b401c68 WatchSource:0}: Error finding container 2eb7ea341b59dd429acd497b228237f119800b8e54ab2c10ecb1d4355b401c68: Status 404 returned error can't find the container with id 2eb7ea341b59dd429acd497b228237f119800b8e54ab2c10ecb1d4355b401c68 Jul 21 19:37:19.892977 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.892952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:19.893064 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:19.892990 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:19.893110 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.893087 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:19.893142 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.893106 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 19:37:19.893142 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.893127 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 19:37:19.893142 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.893129 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs podName:30e72480-263d-4449-87d6-4a30fdbcfcee nodeName:}" failed. No retries permitted until 2026-07-21 19:37:20.893116071 +0000 UTC m=+4.077557735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs") pod "network-metrics-daemon-fjfkl" (UID: "30e72480-263d-4449-87d6-4a30fdbcfcee") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:19.893142 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.893139 2557 projected.go:196] Error preparing data for projected volume kube-api-access-llfv6 for pod openshift-network-diagnostics/network-check-target-ll8xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:19.893277 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:19.893182 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6 podName:18203bb5-2bdc-42cb-ad2f-bfce2b2edd39 nodeName:}" failed. No retries permitted until 2026-07-21 19:37:20.893165785 +0000 UTC m=+4.077607446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-llfv6" (UniqueName: "kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6") pod "network-check-target-ll8xj" (UID: "18203bb5-2bdc-42cb-ad2f-bfce2b2edd39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:19.900688 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:19.900654 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462d9c54_6e71_424a_9489_166c81de167a.slice/crio-49a052a14c646095cec10f14208cfd0138c69177ae5bf9ae316fd88d5d352c46 WatchSource:0}: Error finding container 49a052a14c646095cec10f14208cfd0138c69177ae5bf9ae316fd88d5d352c46: Status 404 returned error can't find the container with id 49a052a14c646095cec10f14208cfd0138c69177ae5bf9ae316fd88d5d352c46 Jul 21 19:37:19.901488 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:19.901461 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265c7b96_c23f_4084_8ff8_fa2789f82cee.slice/crio-004c469a4697a78ea85a8ec84b40ac66d12d973a626588536ee96d4fec6d3809 WatchSource:0}: Error finding container 004c469a4697a78ea85a8ec84b40ac66d12d973a626588536ee96d4fec6d3809: Status 404 returned error can't find the container with id 004c469a4697a78ea85a8ec84b40ac66d12d973a626588536ee96d4fec6d3809 Jul 21 19:37:19.923692 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:19.923661 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96f04b2_899b_4860_8654_6e0061a4d0dc.slice/crio-139cf5b36b93141f1aaa553c676a359c3f71420e43caa594e03ef158690fa3a2 WatchSource:0}: Error finding container 139cf5b36b93141f1aaa553c676a359c3f71420e43caa594e03ef158690fa3a2: Status 404 returned error can't find the container with id 139cf5b36b93141f1aaa553c676a359c3f71420e43caa594e03ef158690fa3a2 Jul 21 19:37:20.005247 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:20.005223 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3f4244_32a9_467b_b77c_26efe45e6aac.slice/crio-e432212b6b456cf03724e87dc5771daa9070b8629d2ccc888a076ac407da9fa4 WatchSource:0}: Error finding container e432212b6b456cf03724e87dc5771daa9070b8629d2ccc888a076ac407da9fa4: Status 404 returned error can't find the container with id e432212b6b456cf03724e87dc5771daa9070b8629d2ccc888a076ac407da9fa4 Jul 21 19:37:20.034239 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:20.034216 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2572ae_c8ef_469f_bec7_82562ab2cd4a.slice/crio-f7ff133be5e2d63c641dd2408049cedff63df7e6ebf547771b97aac3526d5898 WatchSource:0}: Error finding container f7ff133be5e2d63c641dd2408049cedff63df7e6ebf547771b97aac3526d5898: Status 404 returned error can't find the container with id f7ff133be5e2d63c641dd2408049cedff63df7e6ebf547771b97aac3526d5898 Jul 21 19:37:20.149904 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.149836 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 19:32:20 +0000 UTC" deadline="2028-01-02 15:49:43.737495434 +0000 UTC" Jul 21 19:37:20.149904 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.149886 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12716h12m23.587622452s" Jul 21 19:37:20.501908 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.501871 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:20.502845 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.502031 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:20.502845 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.502549 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:20.502845 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.502658 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:20.512380 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.512325 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-64pkq" event={"ID":"0e3f4244-32a9-467b-b77c-26efe45e6aac","Type":"ContainerStarted","Data":"e432212b6b456cf03724e87dc5771daa9070b8629d2ccc888a076ac407da9fa4"} Jul 21 19:37:20.521076 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.521040 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x26r8" event={"ID":"265c7b96-c23f-4084-8ff8-fa2789f82cee","Type":"ContainerStarted","Data":"004c469a4697a78ea85a8ec84b40ac66d12d973a626588536ee96d4fec6d3809"} Jul 21 19:37:20.527129 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.527093 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-djwk4" event={"ID":"02c2a5d8-4576-45b1-9c2a-ddbd5012322b","Type":"ContainerStarted","Data":"f5cbf693c4b1dd78c0ee849bd41c9edc9557ba8e941538aec918ce8552f16607"} Jul 21 19:37:20.541577 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.541540 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k99vr" event={"ID":"e8cdaab2-481b-43f1-b75d-45e7cd7e7824","Type":"ContainerStarted","Data":"ada205e316d0324f7adcf038d5e4e00194eaf4a513690386a9f571df69d5a318"} Jul 21 19:37:20.568297 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.568260 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" event={"ID":"008bd4e4f63a8822603b78b98862ee55","Type":"ContainerStarted","Data":"698b8ddc3b3a384344257e11d1ea5d18dd8c930f1ce0450f3cb9f53b6194fbd5"} Jul 21 19:37:20.574444 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.574411 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" event={"ID":"4d2572ae-c8ef-469f-bec7-82562ab2cd4a","Type":"ContainerStarted","Data":"f7ff133be5e2d63c641dd2408049cedff63df7e6ebf547771b97aac3526d5898"} Jul 21 19:37:20.586193 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.586158 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jqq6h" event={"ID":"b96f04b2-899b-4860-8654-6e0061a4d0dc","Type":"ContainerStarted","Data":"139cf5b36b93141f1aaa553c676a359c3f71420e43caa594e03ef158690fa3a2"} Jul 21 19:37:20.602002 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.601967 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" event={"ID":"462d9c54-6e71-424a-9489-166c81de167a","Type":"ContainerStarted","Data":"49a052a14c646095cec10f14208cfd0138c69177ae5bf9ae316fd88d5d352c46"} Jul 21 19:37:20.625359 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.625223 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" event={"ID":"66af5356-d22c-46a3-972b-76f5050acfee","Type":"ContainerStarted","Data":"2eb7ea341b59dd429acd497b228237f119800b8e54ab2c10ecb1d4355b401c68"} Jul 21 19:37:20.643868 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.643824 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" event={"ID":"461a408f-8393-47e7-96d2-a8748a4121b1","Type":"ContainerStarted","Data":"8969dfde924b28676c6fa8ea92278b4a6aa80edd04a298d3294d4d8ae05c8f05"} Jul 21 19:37:20.652289 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.652220 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" event={"ID":"bfae3c7bef61f9e9cdfc141005768ce6","Type":"ContainerStarted","Data":"e65fbb09470e31015054090079e9be22faf0a3cff01dbecf7c278d8221b27f1d"} Jul 21 19:37:20.800592 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.800498 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:20.800777 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.800718 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:20.800841 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.800802 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret podName:aba01382-69a9-4443-b1a2-fe15605e698d nodeName:}" failed. No retries permitted until 2026-07-21 19:37:22.800782231 +0000 UTC m=+5.985223909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret") pod "global-pull-secret-syncer-kd25k" (UID: "aba01382-69a9-4443-b1a2-fe15605e698d") : object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:20.902476 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.901573 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:20.902476 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:20.901638 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:20.902476 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.901812 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:20.902476 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.901878 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs podName:30e72480-263d-4449-87d6-4a30fdbcfcee nodeName:}" failed. No retries permitted until 2026-07-21 19:37:22.901856619 +0000 UTC m=+6.086298283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs") pod "network-metrics-daemon-fjfkl" (UID: "30e72480-263d-4449-87d6-4a30fdbcfcee") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:20.902476 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.902334 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 19:37:20.902476 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.902358 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 19:37:20.902476 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.902374 2557 projected.go:196] Error preparing data for projected volume kube-api-access-llfv6 for pod openshift-network-diagnostics/network-check-target-ll8xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:20.902476 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:20.902424 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6 podName:18203bb5-2bdc-42cb-ad2f-bfce2b2edd39 nodeName:}" failed. No retries permitted until 2026-07-21 19:37:22.902407163 +0000 UTC m=+6.086848827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-llfv6" (UniqueName: "kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6") pod "network-check-target-ll8xj" (UID: "18203bb5-2bdc-42cb-ad2f-bfce2b2edd39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:21.151161 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:21.151053 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 19:32:20 +0000 UTC" deadline="2028-04-12 12:07:31.065512596 +0000 UTC" Jul 21 19:37:21.151161 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:21.151117 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15136h30m9.914400017s" Jul 21 19:37:21.499566 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:21.499479 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:21.499740 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:21.499636 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:22.507914 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:22.507601 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:22.507914 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.507729 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:22.507914 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:22.507767 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:22.507914 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.507842 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:22.818096 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:22.817998 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:22.818281 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.818182 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:22.818281 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.818253 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret podName:aba01382-69a9-4443-b1a2-fe15605e698d nodeName:}" failed. No retries permitted until 2026-07-21 19:37:26.818232239 +0000 UTC m=+10.002673924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret") pod "global-pull-secret-syncer-kd25k" (UID: "aba01382-69a9-4443-b1a2-fe15605e698d") : object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:22.918762 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:22.918722 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:22.918979 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:22.918789 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:22.919059 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.918973 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:22.919059 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.919045 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs podName:30e72480-263d-4449-87d6-4a30fdbcfcee nodeName:}" failed. No retries permitted until 2026-07-21 19:37:26.919023899 +0000 UTC m=+10.103465563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs") pod "network-metrics-daemon-fjfkl" (UID: "30e72480-263d-4449-87d6-4a30fdbcfcee") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:22.919496 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.919472 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 19:37:22.919596 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.919505 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 19:37:22.919596 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.919521 2557 projected.go:196] Error preparing data for projected volume kube-api-access-llfv6 for pod openshift-network-diagnostics/network-check-target-ll8xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:22.919596 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:22.919569 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6 podName:18203bb5-2bdc-42cb-ad2f-bfce2b2edd39 nodeName:}" failed. No retries permitted until 2026-07-21 19:37:26.919553891 +0000 UTC m=+10.103995558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-llfv6" (UniqueName: "kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6") pod "network-check-target-ll8xj" (UID: "18203bb5-2bdc-42cb-ad2f-bfce2b2edd39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:23.499328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:23.499285 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:23.499521 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:23.499438 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:24.503860 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:24.503817 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:24.504330 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:24.503819 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:24.504330 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:24.503973 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:24.504330 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:24.504048 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:25.499489 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:25.499382 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:25.499648 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:25.499568 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:26.499933 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:26.499897 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:26.500432 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.500058 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:26.500685 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:26.500532 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:26.500685 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.500643 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:26.855846 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:26.855386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:26.855846 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.855536 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:26.855846 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.855610 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret podName:aba01382-69a9-4443-b1a2-fe15605e698d nodeName:}" failed. No retries permitted until 2026-07-21 19:37:34.855589055 +0000 UTC m=+18.040030720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret") pod "global-pull-secret-syncer-kd25k" (UID: "aba01382-69a9-4443-b1a2-fe15605e698d") : object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:26.956666 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:26.956624 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:26.956855 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:26.956700 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:26.956855 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.956789 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 19:37:26.956855 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.956822 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 19:37:26.956855 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.956837 2557 projected.go:196] Error preparing data for projected volume kube-api-access-llfv6 for pod openshift-network-diagnostics/network-check-target-ll8xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:26.957117 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.956894 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:26.957117 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.956920 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6 podName:18203bb5-2bdc-42cb-ad2f-bfce2b2edd39 nodeName:}" failed. No retries permitted until 2026-07-21 19:37:34.95687901 +0000 UTC m=+18.141320674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-llfv6" (UniqueName: "kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6") pod "network-check-target-ll8xj" (UID: "18203bb5-2bdc-42cb-ad2f-bfce2b2edd39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:26.957117 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:26.957024 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs podName:30e72480-263d-4449-87d6-4a30fdbcfcee nodeName:}" failed. No retries permitted until 2026-07-21 19:37:34.957004602 +0000 UTC m=+18.141446268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs") pod "network-metrics-daemon-fjfkl" (UID: "30e72480-263d-4449-87d6-4a30fdbcfcee") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:27.499128 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:27.499086 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:27.499337 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:27.499235 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:28.500014 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:28.499979 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:28.500517 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:28.500110 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:28.500517 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:28.499979 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:28.500517 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:28.500226 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:29.499076 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:29.499025 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:29.499377 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:29.499209 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:30.498857 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:30.498824 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:30.499303 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:30.498982 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:30.499303 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:30.498824 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:30.499303 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:30.499254 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:31.498907 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:31.498856 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:31.499343 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:31.499002 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:32.499099 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:32.499065 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:32.499517 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:32.499076 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:32.499517 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:32.499182 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:32.499517 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:32.499276 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:33.499382 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:33.499346 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:33.499857 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:33.499498 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:34.499651 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:34.499615 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:34.500133 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:34.499758 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:34.500133 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:34.499813 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:34.500133 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:34.499944 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:34.916164 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:34.916124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:34.916315 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:34.916254 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:34.916401 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:34.916326 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret podName:aba01382-69a9-4443-b1a2-fe15605e698d nodeName:}" failed. No retries permitted until 2026-07-21 19:37:50.916309374 +0000 UTC m=+34.100751033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret") pod "global-pull-secret-syncer-kd25k" (UID: "aba01382-69a9-4443-b1a2-fe15605e698d") : object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:35.017016 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:35.016973 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:35.017211 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:35.017064 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:35.017211 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:35.017177 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:35.017211 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:35.017202 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 19:37:35.017376 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:35.017264 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs podName:30e72480-263d-4449-87d6-4a30fdbcfcee nodeName:}" failed. No retries permitted until 2026-07-21 19:37:51.017239377 +0000 UTC m=+34.201681035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs") pod "network-metrics-daemon-fjfkl" (UID: "30e72480-263d-4449-87d6-4a30fdbcfcee") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:35.017787 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:35.017762 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 19:37:35.017787 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:35.017787 2557 projected.go:196] Error preparing data for projected volume kube-api-access-llfv6 for pod openshift-network-diagnostics/network-check-target-ll8xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:35.017995 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:35.017845 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6 podName:18203bb5-2bdc-42cb-ad2f-bfce2b2edd39 nodeName:}" failed. No retries permitted until 2026-07-21 19:37:51.017829518 +0000 UTC m=+34.202271176 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-llfv6" (UniqueName: "kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6") pod "network-check-target-ll8xj" (UID: "18203bb5-2bdc-42cb-ad2f-bfce2b2edd39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:35.499460 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:35.499425 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:35.499652 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:35.499550 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:36.499139 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:36.499104 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:36.499594 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:36.499218 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:36.499594 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:36.499266 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:36.499594 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:36.499354 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:37.498834 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:37.498789 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:37.499016 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:37.498936 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:38.499787 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:38.499755 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:38.500224 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:38.499854 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:38.500224 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:38.499908 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:38.500224 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:38.500031 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:39.499318 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:39.499288 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:39.499500 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:39.499421 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:40.499241 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:40.499203 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:40.499868 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:40.499345 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:40.499868 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:40.499405 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:40.499868 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:40.499491 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:41.500069 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.499766 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:41.500696 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:41.500157 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:41.704573 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.704536 2557 generic.go:356] "Generic (PLEG): container finished" podID="66af5356-d22c-46a3-972b-76f5050acfee" containerID="64a49d48084660df933a5c3cad56c687ffa9f3e563db4f35616a2227ea324cde" exitCode=0 Jul 21 19:37:41.704818 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.704626 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" event={"ID":"66af5356-d22c-46a3-972b-76f5050acfee","Type":"ContainerDied","Data":"64a49d48084660df933a5c3cad56c687ffa9f3e563db4f35616a2227ea324cde"} Jul 21 19:37:41.706412 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.706376 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" event={"ID":"461a408f-8393-47e7-96d2-a8748a4121b1","Type":"ContainerStarted","Data":"9f0a74e96918a567ac2db4290117a8ef56ebabf700b4ab4f7849983d553d607f"} Jul 21 19:37:41.707891 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.707851 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" event={"ID":"bfae3c7bef61f9e9cdfc141005768ce6","Type":"ContainerStarted","Data":"92cb42fac643075b9b64cba7b4e7b21166fec64af939080999b569dbdb50a809"} Jul 21 19:37:41.709524 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.709502 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-64pkq" event={"ID":"0e3f4244-32a9-467b-b77c-26efe45e6aac","Type":"ContainerStarted","Data":"982337453b20adf7b45fb14811c5c7d3de483310ec6db64863a7ae187f3ecf1f"} Jul 21 19:37:41.711292 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.711261 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x26r8" event={"ID":"265c7b96-c23f-4084-8ff8-fa2789f82cee","Type":"ContainerStarted","Data":"2aa457828d11ea5beb4866f433dab486fda5066059f011333597e460e82ca871"} Jul 21 19:37:41.712760 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.712737 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-djwk4" event={"ID":"02c2a5d8-4576-45b1-9c2a-ddbd5012322b","Type":"ContainerStarted","Data":"8a2fd6996da5ce4906c0a5a068950a2c0cd0544f66f3432c156c0f0527c8cf0d"} Jul 21 19:37:41.714291 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.714271 2557 generic.go:356] "Generic (PLEG): container finished" podID="008bd4e4f63a8822603b78b98862ee55" containerID="66a891d90c0ff6fd20f869884a524b352d6656806b91114ccca45b28bb198c37" exitCode=0 Jul 21 19:37:41.714382 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.714293 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" event={"ID":"008bd4e4f63a8822603b78b98862ee55","Type":"ContainerDied","Data":"66a891d90c0ff6fd20f869884a524b352d6656806b91114ccca45b28bb198c37"} Jul 21 19:37:41.715856 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.715833 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" event={"ID":"4d2572ae-c8ef-469f-bec7-82562ab2cd4a","Type":"ContainerStarted","Data":"11ccc131361714bbd215da8f72e2e29f36968e2fce819e787c97a7858c9d8ea9"} Jul 21 19:37:41.717621 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.717596 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jqq6h" event={"ID":"b96f04b2-899b-4860-8654-6e0061a4d0dc","Type":"ContainerStarted","Data":"6efb00bebd70fcfc209dd3550aef2feb44d93084ff6ab9f762c098f7e945a015"} Jul 21 19:37:41.720764 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.720740 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" event={"ID":"462d9c54-6e71-424a-9489-166c81de167a","Type":"ContainerStarted","Data":"cee69339eec09d6e22c4b7e476546dddcbb894329f6c49bcc54f9b060ded1060"} Jul 21 19:37:41.720859 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.720796 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" event={"ID":"462d9c54-6e71-424a-9489-166c81de167a","Type":"ContainerStarted","Data":"cb3a9ee23cd15f58562e4671ad265bbb72571059fdff012dc3e597a166295ca2"} Jul 21 19:37:41.720859 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.720826 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" event={"ID":"462d9c54-6e71-424a-9489-166c81de167a","Type":"ContainerStarted","Data":"259cd90453bb748713fde7810cc215bd83f929421d315b1de8cb78f5e514ec34"} Jul 21 19:37:41.720992 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.720862 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" event={"ID":"462d9c54-6e71-424a-9489-166c81de167a","Type":"ContainerStarted","Data":"470d352fc9e085bd374615c40160c08a7644dff2752955dfc7d4be2ed4c0ee3f"} Jul 21 19:37:41.720992 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.720914 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" event={"ID":"462d9c54-6e71-424a-9489-166c81de167a","Type":"ContainerStarted","Data":"9f26136b71e0d8bf8655878533d91db370b93ff0695545b187e7a5a878146fd4"} Jul 21 19:37:41.720992 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.720949 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" event={"ID":"462d9c54-6e71-424a-9489-166c81de167a","Type":"ContainerStarted","Data":"586e91b1bd865982aeae442d37bbb618decc9473685b6735344cd07059329813"} Jul 21 19:37:41.760588 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.760538 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jqq6h" podStartSLOduration=2.954671004 podStartE2EDuration="23.760517312s" totalImagesPullingTime="20.805846308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:41.744700242 +0000 UTC m=+24.929141924" watchObservedRunningTime="2026-07-21 19:37:41.760517312 +0000 UTC m=+24.944958994" Jul 21 19:37:41.760813 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.760772 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-djwk4" podStartSLOduration=2.901227956 podStartE2EDuration="23.760760511s" totalImagesPullingTime="20.859532555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:41.760628897 +0000 UTC m=+24.945070580" watchObservedRunningTime="2026-07-21 19:37:41.760760511 +0000 UTC m=+24.945202195" Jul 21 19:37:41.782739 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.782538 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m4kjl" podStartSLOduration=3.036715281 podStartE2EDuration="23.782520563s" totalImagesPullingTime="20.745805282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:41.782174594 +0000 UTC m=+24.966616297" watchObservedRunningTime="2026-07-21 19:37:41.782520563 +0000 UTC m=+24.966962247" Jul 21 19:37:41.817434 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.817397 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x26r8" podStartSLOduration=2.938665491 podStartE2EDuration="23.817382619s" totalImagesPullingTime="20.878717128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:41.816814726 +0000 UTC m=+25.001256443" watchObservedRunningTime="2026-07-21 19:37:41.817382619 +0000 UTC m=+25.001824300" Jul 21 19:37:41.839201 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.839154 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-64pkq" podStartSLOduration=3.111677237 podStartE2EDuration="23.839140737s" totalImagesPullingTime="20.7274635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:41.838919751 +0000 UTC m=+25.023361443" watchObservedRunningTime="2026-07-21 19:37:41.839140737 +0000 UTC m=+25.023582417" Jul 21 19:37:41.853082 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:41.853043 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-158.ec2.internal" podStartSLOduration=22.853027782 podStartE2EDuration="22.853027782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:41.852742913 +0000 UTC m=+25.037184577" watchObservedRunningTime="2026-07-21 19:37:41.853027782 +0000 UTC m=+25.037469462" Jul 21 19:37:42.413009 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:42.412977 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 19:37:42.499240 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:42.499203 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:42.499402 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:42.499207 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:42.499402 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:42.499346 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:42.499497 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:42.499412 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:42.724378 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:42.724336 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k99vr" event={"ID":"e8cdaab2-481b-43f1-b75d-45e7cd7e7824","Type":"ContainerStarted","Data":"b8b1120dca8d446b27797794fa7c554ade34626cdefdac26622da37f399e370f"} Jul 21 19:37:42.726432 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:42.726405 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" event={"ID":"008bd4e4f63a8822603b78b98862ee55","Type":"ContainerStarted","Data":"a9df28a68522156c0af4acd731c17541820942c2c0853996452ab49b7437fd5d"} Jul 21 19:37:42.731482 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:42.731455 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" event={"ID":"461a408f-8393-47e7-96d2-a8748a4121b1","Type":"ContainerStarted","Data":"31c4634d4d819b2fc613bda4a4440915773e3ca09c962e6b1b6080db05124d5d"} Jul 21 19:37:42.742529 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:42.742488 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k99vr" podStartSLOduration=3.819010011 podStartE2EDuration="24.742475412s" totalImagesPullingTime="20.923465401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:42.742171971 +0000 UTC m=+25.926613665" watchObservedRunningTime="2026-07-21 19:37:42.742475412 +0000 UTC m=+25.926917093" Jul 21 19:37:42.760296 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:42.760255 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-158.ec2.internal" podStartSLOduration=23.760240957 podStartE2EDuration="23.760240957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:42.759717454 +0000 UTC m=+25.944159136" watchObservedRunningTime="2026-07-21 19:37:42.760240957 +0000 UTC m=+25.944682637" Jul 21 19:37:43.291999 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:43.291864 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T19:37:42.413002053Z","UUID":"3b92e3a5-313e-4670-af27-75b13813d03f","Handler":null,"Name":"","Endpoint":""} Jul 21 19:37:43.293993 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:43.293927 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 19:37:43.293993 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:43.293970 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 19:37:43.499315 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:43.499287 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:43.499456 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:43.499432 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:43.737938 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:43.737903 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" event={"ID":"462d9c54-6e71-424a-9489-166c81de167a","Type":"ContainerStarted","Data":"c3dfa51baffa3bfa5b651aaff315cc9815c09f6213c8816025ba532cf51aadee"} Jul 21 19:37:44.499319 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:44.499285 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:44.499498 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:44.499285 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:44.499498 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:44.499412 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:44.499616 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:44.499495 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:44.742438 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:44.742402 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" event={"ID":"461a408f-8393-47e7-96d2-a8748a4121b1","Type":"ContainerStarted","Data":"256a704be7ef248258b0cb6409edcc04bc3b3ae7c2096f0944d8363e093e8767"} Jul 21 19:37:44.764273 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:44.764181 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hllkq" podStartSLOduration=3.179315989 podStartE2EDuration="26.764164154s" totalImagesPullingTime="23.584848165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:44.764084225 +0000 UTC m=+27.948525929" watchObservedRunningTime="2026-07-21 19:37:44.764164154 +0000 UTC m=+27.948605834" Jul 21 19:37:45.499276 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:45.499204 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:45.499413 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:45.499321 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:46.058189 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.057947 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:46.058727 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.058604 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:46.499272 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.499239 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:46.499411 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.499283 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:46.499411 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:46.499362 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:46.499561 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:46.499477 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:46.748420 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.748383 2557 generic.go:356] "Generic (PLEG): container finished" podID="66af5356-d22c-46a3-972b-76f5050acfee" containerID="e5b112c0a6e40309e52f7e517dff4704666662c302c03479b22df79c877dd174" exitCode=0 Jul 21 19:37:46.748597 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.748467 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" event={"ID":"66af5356-d22c-46a3-972b-76f5050acfee","Type":"ContainerDied","Data":"e5b112c0a6e40309e52f7e517dff4704666662c302c03479b22df79c877dd174"} Jul 21 19:37:46.752300 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.752275 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" event={"ID":"462d9c54-6e71-424a-9489-166c81de167a","Type":"ContainerStarted","Data":"e6b7eb2095f5f41d4ce99f4f1108657e3e888cabb7bcd83590db5b00da21d847"} Jul 21 19:37:46.752631 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.752606 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:46.752700 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.752656 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:46.752788 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.752707 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:46.752788 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.752764 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:46.752974 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.752955 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jqq6h" Jul 21 19:37:46.767425 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.767402 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:46.767547 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:46.767536 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:37:47.499124 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:47.499092 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:47.499703 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:47.499214 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:48.110712 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:48.110462 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" podStartSLOduration=9.169977022 podStartE2EDuration="30.110445167s" totalImagesPullingTime="20.940468145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:46.819991703 +0000 UTC m=+30.004433393" watchObservedRunningTime="2026-07-21 19:37:48.110445167 +0000 UTC m=+31.294886848" Jul 21 19:37:48.111188 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:48.111168 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ll8xj"] Jul 21 19:37:48.111284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:48.111272 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:48.111364 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:48.111349 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:48.114280 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:48.114250 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kd25k"] Jul 21 19:37:48.114402 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:48.114362 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:48.114496 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:48.114465 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:48.115029 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:48.115008 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fjfkl"] Jul 21 19:37:48.115122 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:48.115099 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:48.115201 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:48.115183 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:48.757867 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:48.757833 2557 generic.go:356] "Generic (PLEG): container finished" podID="66af5356-d22c-46a3-972b-76f5050acfee" containerID="f89cefd6b55942c4b84b905f64672e4317528f3e18db2d346989d261c2ca2c2d" exitCode=0 Jul 21 19:37:48.758349 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:48.757921 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" event={"ID":"66af5356-d22c-46a3-972b-76f5050acfee","Type":"ContainerDied","Data":"f89cefd6b55942c4b84b905f64672e4317528f3e18db2d346989d261c2ca2c2d"} Jul 21 19:37:49.499330 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:49.499284 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:49.499480 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:49.499306 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:49.499480 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:49.499416 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:49.499480 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:49.499449 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:49.763921 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:49.763892 2557 generic.go:356] "Generic (PLEG): container finished" podID="66af5356-d22c-46a3-972b-76f5050acfee" containerID="9b3510a00bd6e5c6fbc0d74c6157706bb31b53bf7c47aba532b7494e28a541d6" exitCode=0 Jul 21 19:37:49.764319 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:49.763958 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" event={"ID":"66af5356-d22c-46a3-972b-76f5050acfee","Type":"ContainerDied","Data":"9b3510a00bd6e5c6fbc0d74c6157706bb31b53bf7c47aba532b7494e28a541d6"} Jul 21 19:37:50.499825 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:50.499791 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:50.500033 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:50.499941 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:50.929887 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:50.929830 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:50.930354 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:50.929987 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:50.930354 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:50.930051 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret podName:aba01382-69a9-4443-b1a2-fe15605e698d nodeName:}" failed. No retries permitted until 2026-07-21 19:38:22.930036205 +0000 UTC m=+66.114477864 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret") pod "global-pull-secret-syncer-kd25k" (UID: "aba01382-69a9-4443-b1a2-fe15605e698d") : object "kube-system"/"original-pull-secret" not registered Jul 21 19:37:51.031081 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:51.031044 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:51.031258 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:51.031101 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:51.031258 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:51.031200 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:51.031258 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:51.031213 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 19:37:51.031258 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:51.031232 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 19:37:51.031258 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:51.031244 2557 projected.go:196] Error preparing data for projected volume kube-api-access-llfv6 for pod openshift-network-diagnostics/network-check-target-ll8xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:51.031465 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:51.031276 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs podName:30e72480-263d-4449-87d6-4a30fdbcfcee nodeName:}" failed. No retries permitted until 2026-07-21 19:38:23.031256978 +0000 UTC m=+66.215698648 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs") pod "network-metrics-daemon-fjfkl" (UID: "30e72480-263d-4449-87d6-4a30fdbcfcee") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 19:37:51.031465 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:51.031292 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6 podName:18203bb5-2bdc-42cb-ad2f-bfce2b2edd39 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:23.031285615 +0000 UTC m=+66.215727273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-llfv6" (UniqueName: "kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6") pod "network-check-target-ll8xj" (UID: "18203bb5-2bdc-42cb-ad2f-bfce2b2edd39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 19:37:51.499591 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:51.499555 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:51.499765 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:51.499560 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:51.499765 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:51.499705 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:51.499890 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:51.499808 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:52.499647 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:52.499618 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:52.500088 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:52.499732 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll8xj" podUID="18203bb5-2bdc-42cb-ad2f-bfce2b2edd39" Jul 21 19:37:53.499106 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.499067 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:53.499295 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.499068 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:53.499295 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:53.499202 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd25k" podUID="aba01382-69a9-4443-b1a2-fe15605e698d" Jul 21 19:37:53.499295 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:53.499233 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjfkl" podUID="30e72480-263d-4449-87d6-4a30fdbcfcee" Jul 21 19:37:53.609507 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.609440 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-158.ec2.internal" event="NodeReady" Jul 21 19:37:53.609836 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.609567 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 19:37:53.658822 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.658790 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dhqw9"] Jul 21 19:37:53.682254 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.682226 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-czf26"] Jul 21 19:37:53.682435 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.682376 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.685035 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.685011 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 19:37:53.685035 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.685031 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-46cqs\"" Jul 21 19:37:53.685501 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.685486 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 19:37:53.698536 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.698516 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dhqw9"] Jul 21 19:37:53.698656 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.698566 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czf26"] Jul 21 19:37:53.698656 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.698629 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czf26" Jul 21 19:37:53.701442 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.701423 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zpj99\"" Jul 21 19:37:53.701548 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.701427 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 19:37:53.701548 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.701429 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 19:37:53.701548 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.701500 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 19:37:53.752126 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.752091 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.752126 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.752127 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmgpj\" (UniqueName: \"kubernetes.io/projected/51ed889a-0a84-4523-8c9b-989d2ac767fe-kube-api-access-zmgpj\") pod \"ingress-canary-czf26\" (UID: \"51ed889a-0a84-4523-8c9b-989d2ac767fe\") " pod="openshift-ingress-canary/ingress-canary-czf26" Jul 21 19:37:53.752314 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.752160 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr4vt\" (UniqueName: \"kubernetes.io/projected/2c1b84bc-9f08-49b0-b538-c09d743953bf-kube-api-access-vr4vt\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.752314 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.752206 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51ed889a-0a84-4523-8c9b-989d2ac767fe-cert\") pod \"ingress-canary-czf26\" (UID: \"51ed889a-0a84-4523-8c9b-989d2ac767fe\") " pod="openshift-ingress-canary/ingress-canary-czf26" Jul 21 19:37:53.752314 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.752242 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1b84bc-9f08-49b0-b538-c09d743953bf-tmp-dir\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.752314 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.752265 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1b84bc-9f08-49b0-b538-c09d743953bf-config-volume\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.852765 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.852723 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1b84bc-9f08-49b0-b538-c09d743953bf-config-volume\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.852969 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.852792 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.852969 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.852810 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmgpj\" (UniqueName: \"kubernetes.io/projected/51ed889a-0a84-4523-8c9b-989d2ac767fe-kube-api-access-zmgpj\") pod \"ingress-canary-czf26\" (UID: \"51ed889a-0a84-4523-8c9b-989d2ac767fe\") " pod="openshift-ingress-canary/ingress-canary-czf26" Jul 21 19:37:53.852969 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.852836 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr4vt\" (UniqueName: \"kubernetes.io/projected/2c1b84bc-9f08-49b0-b538-c09d743953bf-kube-api-access-vr4vt\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.852969 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.852853 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51ed889a-0a84-4523-8c9b-989d2ac767fe-cert\") pod \"ingress-canary-czf26\" (UID: \"51ed889a-0a84-4523-8c9b-989d2ac767fe\") " pod="openshift-ingress-canary/ingress-canary-czf26" Jul 21 19:37:53.852969 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:53.852965 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 19:37:53.853236 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.853015 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1b84bc-9f08-49b0-b538-c09d743953bf-tmp-dir\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.853236 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:53.853028 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls podName:2c1b84bc-9f08-49b0-b538-c09d743953bf nodeName:}" failed. No retries permitted until 2026-07-21 19:37:54.353009806 +0000 UTC m=+37.537451476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls") pod "dns-default-dhqw9" (UID: "2c1b84bc-9f08-49b0-b538-c09d743953bf") : secret "dns-default-metrics-tls" not found Jul 21 19:37:53.853408 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.853270 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1b84bc-9f08-49b0-b538-c09d743953bf-tmp-dir\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.853408 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.853386 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1b84bc-9f08-49b0-b538-c09d743953bf-config-volume\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.857119 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.857093 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51ed889a-0a84-4523-8c9b-989d2ac767fe-cert\") pod \"ingress-canary-czf26\" (UID: \"51ed889a-0a84-4523-8c9b-989d2ac767fe\") " pod="openshift-ingress-canary/ingress-canary-czf26" Jul 21 19:37:53.863337 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.863175 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr4vt\" (UniqueName: \"kubernetes.io/projected/2c1b84bc-9f08-49b0-b538-c09d743953bf-kube-api-access-vr4vt\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:53.863481 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:53.863461 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmgpj\" (UniqueName: \"kubernetes.io/projected/51ed889a-0a84-4523-8c9b-989d2ac767fe-kube-api-access-zmgpj\") pod \"ingress-canary-czf26\" (UID: \"51ed889a-0a84-4523-8c9b-989d2ac767fe\") " pod="openshift-ingress-canary/ingress-canary-czf26" Jul 21 19:37:54.007228 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.007192 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czf26" Jul 21 19:37:54.237481 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.237452 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czf26"] Jul 21 19:37:54.241002 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:37:54.240976 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ed889a_0a84_4523_8c9b_989d2ac767fe.slice/crio-0b87d8fdb79e7979425ff124c3a4afc2474be44c6d3c724ae40f39c1961c5465 WatchSource:0}: Error finding container 0b87d8fdb79e7979425ff124c3a4afc2474be44c6d3c724ae40f39c1961c5465: Status 404 returned error can't find the container with id 0b87d8fdb79e7979425ff124c3a4afc2474be44c6d3c724ae40f39c1961c5465 Jul 21 19:37:54.357598 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.357565 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:54.357731 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:54.357712 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 19:37:54.357790 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:54.357781 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls podName:2c1b84bc-9f08-49b0-b538-c09d743953bf nodeName:}" failed. No retries permitted until 2026-07-21 19:37:55.357765235 +0000 UTC m=+38.542206899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls") pod "dns-default-dhqw9" (UID: "2c1b84bc-9f08-49b0-b538-c09d743953bf") : secret "dns-default-metrics-tls" not found Jul 21 19:37:54.499680 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.499648 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:37:54.502455 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.502433 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nmxjk\"" Jul 21 19:37:54.502602 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.502460 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 19:37:54.502602 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.502521 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 19:37:54.778938 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.778838 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czf26" event={"ID":"51ed889a-0a84-4523-8c9b-989d2ac767fe","Type":"ContainerStarted","Data":"0b87d8fdb79e7979425ff124c3a4afc2474be44c6d3c724ae40f39c1961c5465"} Jul 21 19:37:54.781963 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.781934 2557 generic.go:356] "Generic (PLEG): container finished" podID="66af5356-d22c-46a3-972b-76f5050acfee" containerID="5e4400bc500a24ae3b1a9bc26b0e5cb253e7028dcf6b39abe4ab938d34409f04" exitCode=0 Jul 21 19:37:54.782108 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:54.781982 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" event={"ID":"66af5356-d22c-46a3-972b-76f5050acfee","Type":"ContainerDied","Data":"5e4400bc500a24ae3b1a9bc26b0e5cb253e7028dcf6b39abe4ab938d34409f04"} Jul 21 19:37:55.365112 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:55.365070 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:55.365289 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:55.365229 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 19:37:55.365370 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:55.365309 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls podName:2c1b84bc-9f08-49b0-b538-c09d743953bf nodeName:}" failed. No retries permitted until 2026-07-21 19:37:57.365288014 +0000 UTC m=+40.549729710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls") pod "dns-default-dhqw9" (UID: "2c1b84bc-9f08-49b0-b538-c09d743953bf") : secret "dns-default-metrics-tls" not found Jul 21 19:37:55.499274 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:55.499235 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:37:55.499437 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:55.499284 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:37:55.502154 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:55.502131 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 19:37:55.502290 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:55.502136 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 19:37:55.502290 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:55.502135 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tcjpf\"" Jul 21 19:37:55.787167 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:55.787136 2557 generic.go:356] "Generic (PLEG): container finished" podID="66af5356-d22c-46a3-972b-76f5050acfee" containerID="58f47bf8cdb995427d3c145cbf41057853c26292c48d756d78713aeaa3afdf25" exitCode=0 Jul 21 19:37:55.787529 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:55.787187 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" event={"ID":"66af5356-d22c-46a3-972b-76f5050acfee","Type":"ContainerDied","Data":"58f47bf8cdb995427d3c145cbf41057853c26292c48d756d78713aeaa3afdf25"} Jul 21 19:37:56.792925 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:56.792753 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" event={"ID":"66af5356-d22c-46a3-972b-76f5050acfee","Type":"ContainerStarted","Data":"e28911e2fbfb4206b95763f1abcf1a51ad5f606f958f0786c7e3a55f46e3da5c"} Jul 21 19:37:56.825280 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:56.825240 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8lbfw" podStartSLOduration=7.829365706 podStartE2EDuration="38.825225666s" totalImagesPullingTime="30.99585996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:56.824043468 +0000 UTC m=+40.008485148" watchObservedRunningTime="2026-07-21 19:37:56.825225666 +0000 UTC m=+40.009667371" Jul 21 19:37:57.381783 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:57.381748 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:37:57.381979 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:57.381918 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 19:37:57.381979 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:37:57.381974 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls podName:2c1b84bc-9f08-49b0-b538-c09d743953bf nodeName:}" failed. No retries permitted until 2026-07-21 19:38:01.381960117 +0000 UTC m=+44.566401778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls") pod "dns-default-dhqw9" (UID: "2c1b84bc-9f08-49b0-b538-c09d743953bf") : secret "dns-default-metrics-tls" not found Jul 21 19:37:57.797035 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:57.796992 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czf26" event={"ID":"51ed889a-0a84-4523-8c9b-989d2ac767fe","Type":"ContainerStarted","Data":"f368c89c3bc8a3a568136d6839bd7d8dd4bcba94d2fb95594a72f9f2c1ab3897"} Jul 21 19:37:57.817423 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:37:57.817373 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-czf26" podStartSLOduration=2.238771684 podStartE2EDuration="4.817358593s" totalImagesPullingTime="2.578586909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:37:57.816604939 +0000 UTC m=+41.001046639" watchObservedRunningTime="2026-07-21 19:37:57.817358593 +0000 UTC m=+41.001800256" Jul 21 19:38:01.411942 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:01.411897 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:38:01.412354 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:01.411999 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 19:38:01.412354 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:01.412066 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls podName:2c1b84bc-9f08-49b0-b538-c09d743953bf nodeName:}" failed. No retries permitted until 2026-07-21 19:38:09.412048196 +0000 UTC m=+52.596489876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls") pod "dns-default-dhqw9" (UID: "2c1b84bc-9f08-49b0-b538-c09d743953bf") : secret "dns-default-metrics-tls" not found Jul 21 19:38:09.466562 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:09.466518 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:38:09.467045 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:09.466645 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 19:38:09.467045 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:09.466698 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls podName:2c1b84bc-9f08-49b0-b538-c09d743953bf nodeName:}" failed. No retries permitted until 2026-07-21 19:38:25.466684994 +0000 UTC m=+68.651126656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls") pod "dns-default-dhqw9" (UID: "2c1b84bc-9f08-49b0-b538-c09d743953bf") : secret "dns-default-metrics-tls" not found Jul 21 19:38:18.772040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:18.772006 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn8h5" Jul 21 19:38:22.897344 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.897311 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5"] Jul 21 19:38:22.899969 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.899954 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-fvv9d"] Jul 21 19:38:22.900109 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.900093 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:22.901328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.901310 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:22.902723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.902705 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 21 19:38:22.902859 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.902795 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 21 19:38:22.902859 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.902803 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 21 19:38:22.902977 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.902857 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t2kzh\"" Jul 21 19:38:22.903667 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.903652 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 21 19:38:22.904141 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.904126 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 21 19:38:22.904275 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.904250 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 21 19:38:22.904377 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.904332 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 21 19:38:22.904669 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.904655 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9w6ng\"" Jul 21 19:38:22.910327 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.910306 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 21 19:38:22.913342 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.913321 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-fvv9d"] Jul 21 19:38:22.914225 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.914201 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5"] Jul 21 19:38:22.966049 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.966022 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:38:22.968883 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.968857 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 19:38:22.980414 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.980396 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aba01382-69a9-4443-b1a2-fe15605e698d-original-pull-secret\") pod \"global-pull-secret-syncer-kd25k\" (UID: \"aba01382-69a9-4443-b1a2-fe15605e698d\") " pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:38:22.997779 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:22.997754 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kcl4p"] Jul 21 19:38:23.001456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.000126 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.003765 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.003747 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ltf9r\"" Jul 21 19:38:23.003858 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.003842 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 19:38:23.004023 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.004003 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 19:38:23.004081 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.004063 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 19:38:23.004622 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.004608 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 19:38:23.009685 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.009667 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 19:38:23.014591 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.014571 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kcl4p"] Jul 21 19:38:23.067124 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.067095 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z67k6\" (UniqueName: \"kubernetes.io/projected/639481df-2652-4b1d-b003-c6aef4eb44af-kube-api-access-z67k6\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.067271 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.067132 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:23.067271 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.067153 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/639481df-2652-4b1d-b003-c6aef4eb44af-trusted-ca\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.067271 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.067175 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/639481df-2652-4b1d-b003-c6aef4eb44af-serving-cert\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.067271 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.067204 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:38:23.067442 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.067387 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:38:23.067442 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.067427 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fncdg\" (UniqueName: \"kubernetes.io/projected/a341677b-760e-4573-b6f0-d90ef217c23e-kube-api-access-fncdg\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:23.067607 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.067451 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/639481df-2652-4b1d-b003-c6aef4eb44af-config\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.070799 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.070780 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 19:38:23.071028 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.071012 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 19:38:23.078150 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:23.078132 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 19:38:23.078213 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:23.078203 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs podName:30e72480-263d-4449-87d6-4a30fdbcfcee nodeName:}" failed. No retries permitted until 2026-07-21 19:39:27.078181753 +0000 UTC m=+130.262623421 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs") pod "network-metrics-daemon-fjfkl" (UID: "30e72480-263d-4449-87d6-4a30fdbcfcee") : secret "metrics-daemon-secret" not found Jul 21 19:38:23.080480 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.080468 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 19:38:23.091282 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.091257 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfv6\" (UniqueName: \"kubernetes.io/projected/18203bb5-2bdc-42cb-ad2f-bfce2b2edd39-kube-api-access-llfv6\") pod \"network-check-target-ll8xj\" (UID: \"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39\") " pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:38:23.110120 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.110089 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd25k" Jul 21 19:38:23.114934 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.114914 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v"] Jul 21 19:38:23.116682 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.116662 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.117983 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.117964 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz"] Jul 21 19:38:23.119475 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.119423 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.120042 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.120022 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 19:38:23.120393 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.120372 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8tqdl\"" Jul 21 19:38:23.121388 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.121368 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 19:38:23.124394 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.124378 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 19:38:23.125032 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.125015 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 19:38:23.125121 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.125063 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 19:38:23.125121 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.125064 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 19:38:23.125736 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.125720 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-thqxg\"" Jul 21 19:38:23.126092 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.126073 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 19:38:23.126243 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.126229 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 19:38:23.137785 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.137768 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz"] Jul 21 19:38:23.150417 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.150353 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v"] Jul 21 19:38:23.168037 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168012 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/639481df-2652-4b1d-b003-c6aef4eb44af-serving-cert\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.168132 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168103 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58826593-b4e6-4152-a0a9-a5b9746af2a9-service-ca-bundle\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.168195 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168135 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fncdg\" (UniqueName: \"kubernetes.io/projected/a341677b-760e-4573-b6f0-d90ef217c23e-kube-api-access-fncdg\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:23.168195 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168159 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/639481df-2652-4b1d-b003-c6aef4eb44af-config\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.168195 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168182 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58826593-b4e6-4152-a0a9-a5b9746af2a9-serving-cert\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.168348 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168311 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58826593-b4e6-4152-a0a9-a5b9746af2a9-tmp\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.168403 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168345 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z67k6\" (UniqueName: \"kubernetes.io/projected/639481df-2652-4b1d-b003-c6aef4eb44af-kube-api-access-z67k6\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.168403 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168368 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58826593-b4e6-4152-a0a9-a5b9746af2a9-snapshots\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.168587 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:23.168566 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 19:38:23.168656 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:23.168637 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls podName:a341677b-760e-4573-b6f0-d90ef217c23e nodeName:}" failed. No retries permitted until 2026-07-21 19:38:23.668616497 +0000 UTC m=+66.853058157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mdjb5" (UID: "a341677b-760e-4573-b6f0-d90ef217c23e") : secret "samples-operator-tls" not found Jul 21 19:38:23.168739 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168703 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:23.168793 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168768 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/639481df-2652-4b1d-b003-c6aef4eb44af-trusted-ca\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.168851 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168826 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58826593-b4e6-4152-a0a9-a5b9746af2a9-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.168928 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168862 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw7nr\" (UniqueName: \"kubernetes.io/projected/58826593-b4e6-4152-a0a9-a5b9746af2a9-kube-api-access-rw7nr\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.168928 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.168901 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/639481df-2652-4b1d-b003-c6aef4eb44af-config\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.169746 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.169726 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/639481df-2652-4b1d-b003-c6aef4eb44af-trusted-ca\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.170367 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.170343 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/639481df-2652-4b1d-b003-c6aef4eb44af-serving-cert\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.193821 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.193777 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z67k6\" (UniqueName: \"kubernetes.io/projected/639481df-2652-4b1d-b003-c6aef4eb44af-kube-api-access-z67k6\") pod \"console-operator-565b7b569d-fvv9d\" (UID: \"639481df-2652-4b1d-b003-c6aef4eb44af\") " pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.194164 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.194074 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fncdg\" (UniqueName: \"kubernetes.io/projected/a341677b-760e-4573-b6f0-d90ef217c23e-kube-api-access-fncdg\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:23.216065 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.216040 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:23.239856 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.239813 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kd25k"] Jul 21 19:38:23.241817 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:23.241786 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba01382_69a9_4443_b1a2_fe15605e698d.slice/crio-2b82e3eab84d502fb7e9129f1b33b5de1a999e1eea448af55493a1df61c14349 WatchSource:0}: Error finding container 2b82e3eab84d502fb7e9129f1b33b5de1a999e1eea448af55493a1df61c14349: Status 404 returned error can't find the container with id 2b82e3eab84d502fb7e9129f1b33b5de1a999e1eea448af55493a1df61c14349 Jul 21 19:38:23.270596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270540 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58826593-b4e6-4152-a0a9-a5b9746af2a9-service-ca-bundle\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.270596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270585 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4272620-b668-47a1-b43d-ba5fd58231be-config\") pod \"service-ca-operator-5bcbc94c78-96z7v\" (UID: \"d4272620-b668-47a1-b43d-ba5fd58231be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.270766 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270619 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58826593-b4e6-4152-a0a9-a5b9746af2a9-serving-cert\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.270766 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270644 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac97b694-bcc5-47c3-a779-9bf9b79bbdf0-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-7k4hz\" (UID: \"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.270766 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270670 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckvn6\" (UniqueName: \"kubernetes.io/projected/ac97b694-bcc5-47c3-a779-9bf9b79bbdf0-kube-api-access-ckvn6\") pod \"kube-storage-version-migrator-operator-7658d9b749-7k4hz\" (UID: \"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.270766 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270697 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4272620-b668-47a1-b43d-ba5fd58231be-serving-cert\") pod \"service-ca-operator-5bcbc94c78-96z7v\" (UID: \"d4272620-b668-47a1-b43d-ba5fd58231be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.270766 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270728 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac97b694-bcc5-47c3-a779-9bf9b79bbdf0-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-7k4hz\" (UID: \"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.270766 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270764 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58826593-b4e6-4152-a0a9-a5b9746af2a9-tmp\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.271113 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270795 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58826593-b4e6-4152-a0a9-a5b9746af2a9-snapshots\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.271113 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270856 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58826593-b4e6-4152-a0a9-a5b9746af2a9-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.271113 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270900 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw7nr\" (UniqueName: \"kubernetes.io/projected/58826593-b4e6-4152-a0a9-a5b9746af2a9-kube-api-access-rw7nr\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.271113 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.270926 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwrj\" (UniqueName: \"kubernetes.io/projected/d4272620-b668-47a1-b43d-ba5fd58231be-kube-api-access-tkwrj\") pod \"service-ca-operator-5bcbc94c78-96z7v\" (UID: \"d4272620-b668-47a1-b43d-ba5fd58231be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.271423 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.271380 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58826593-b4e6-4152-a0a9-a5b9746af2a9-service-ca-bundle\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.271582 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.271555 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58826593-b4e6-4152-a0a9-a5b9746af2a9-tmp\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.271678 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.271606 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58826593-b4e6-4152-a0a9-a5b9746af2a9-snapshots\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.272325 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.272301 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58826593-b4e6-4152-a0a9-a5b9746af2a9-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.273255 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.273233 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58826593-b4e6-4152-a0a9-a5b9746af2a9-serving-cert\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.279781 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.279761 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw7nr\" (UniqueName: \"kubernetes.io/projected/58826593-b4e6-4152-a0a9-a5b9746af2a9-kube-api-access-rw7nr\") pod \"insights-operator-555d9d7885-kcl4p\" (UID: \"58826593-b4e6-4152-a0a9-a5b9746af2a9\") " pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.310244 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.310221 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kcl4p" Jul 21 19:38:23.310757 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.310738 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nmxjk\"" Jul 21 19:38:23.319128 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.319109 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:38:23.339903 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.339851 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-fvv9d"] Jul 21 19:38:23.342658 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:23.342633 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639481df_2652_4b1d_b003_c6aef4eb44af.slice/crio-356cc6f3e49937688d8741b5d1cefb3ac2e6fce4a59e9cf8a51b9ced8444dd98 WatchSource:0}: Error finding container 356cc6f3e49937688d8741b5d1cefb3ac2e6fce4a59e9cf8a51b9ced8444dd98: Status 404 returned error can't find the container with id 356cc6f3e49937688d8741b5d1cefb3ac2e6fce4a59e9cf8a51b9ced8444dd98 Jul 21 19:38:23.372687 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.372399 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4272620-b668-47a1-b43d-ba5fd58231be-config\") pod \"service-ca-operator-5bcbc94c78-96z7v\" (UID: \"d4272620-b668-47a1-b43d-ba5fd58231be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.372687 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.372444 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac97b694-bcc5-47c3-a779-9bf9b79bbdf0-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-7k4hz\" (UID: \"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.372687 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.372478 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckvn6\" (UniqueName: \"kubernetes.io/projected/ac97b694-bcc5-47c3-a779-9bf9b79bbdf0-kube-api-access-ckvn6\") pod \"kube-storage-version-migrator-operator-7658d9b749-7k4hz\" (UID: \"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.372687 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.372506 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4272620-b668-47a1-b43d-ba5fd58231be-serving-cert\") pod \"service-ca-operator-5bcbc94c78-96z7v\" (UID: \"d4272620-b668-47a1-b43d-ba5fd58231be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.372687 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.372544 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac97b694-bcc5-47c3-a779-9bf9b79bbdf0-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-7k4hz\" (UID: \"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.372687 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.372624 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwrj\" (UniqueName: \"kubernetes.io/projected/d4272620-b668-47a1-b43d-ba5fd58231be-kube-api-access-tkwrj\") pod \"service-ca-operator-5bcbc94c78-96z7v\" (UID: \"d4272620-b668-47a1-b43d-ba5fd58231be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.373613 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.373584 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4272620-b668-47a1-b43d-ba5fd58231be-config\") pod \"service-ca-operator-5bcbc94c78-96z7v\" (UID: \"d4272620-b668-47a1-b43d-ba5fd58231be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.374200 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.374176 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac97b694-bcc5-47c3-a779-9bf9b79bbdf0-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-7k4hz\" (UID: \"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.389538 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.378998 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac97b694-bcc5-47c3-a779-9bf9b79bbdf0-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-7k4hz\" (UID: \"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.391043 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.390674 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4272620-b668-47a1-b43d-ba5fd58231be-serving-cert\") pod \"service-ca-operator-5bcbc94c78-96z7v\" (UID: \"d4272620-b668-47a1-b43d-ba5fd58231be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.394529 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.394454 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwrj\" (UniqueName: \"kubernetes.io/projected/d4272620-b668-47a1-b43d-ba5fd58231be-kube-api-access-tkwrj\") pod \"service-ca-operator-5bcbc94c78-96z7v\" (UID: \"d4272620-b668-47a1-b43d-ba5fd58231be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.394837 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.394817 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckvn6\" (UniqueName: \"kubernetes.io/projected/ac97b694-bcc5-47c3-a779-9bf9b79bbdf0-kube-api-access-ckvn6\") pod \"kube-storage-version-migrator-operator-7658d9b749-7k4hz\" (UID: \"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.433315 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.433287 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" Jul 21 19:38:23.439129 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.439092 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" Jul 21 19:38:23.477452 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.477385 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kcl4p"] Jul 21 19:38:23.481209 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:23.481177 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58826593_b4e6_4152_a0a9_a5b9746af2a9.slice/crio-93bff6db9c2533a94e52092446e685074a361ab8afe261b9a4bec8518fa475c3 WatchSource:0}: Error finding container 93bff6db9c2533a94e52092446e685074a361ab8afe261b9a4bec8518fa475c3: Status 404 returned error can't find the container with id 93bff6db9c2533a94e52092446e685074a361ab8afe261b9a4bec8518fa475c3 Jul 21 19:38:23.492144 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.492076 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ll8xj"] Jul 21 19:38:23.497259 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:23.497231 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18203bb5_2bdc_42cb_ad2f_bfce2b2edd39.slice/crio-312578d4463bc2f7e6dd9e34bd2d2a3d4965046981b99cc5a8c921e0904c9d7e WatchSource:0}: Error finding container 312578d4463bc2f7e6dd9e34bd2d2a3d4965046981b99cc5a8c921e0904c9d7e: Status 404 returned error can't find the container with id 312578d4463bc2f7e6dd9e34bd2d2a3d4965046981b99cc5a8c921e0904c9d7e Jul 21 19:38:23.574673 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.574652 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v"] Jul 21 19:38:23.576971 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:23.576937 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4272620_b668_47a1_b43d_ba5fd58231be.slice/crio-b9ce3dd077bc4dcd618093c7f5d20dcad6591389c90e89e76ba62f15ea6a51f2 WatchSource:0}: Error finding container b9ce3dd077bc4dcd618093c7f5d20dcad6591389c90e89e76ba62f15ea6a51f2: Status 404 returned error can't find the container with id b9ce3dd077bc4dcd618093c7f5d20dcad6591389c90e89e76ba62f15ea6a51f2 Jul 21 19:38:23.591469 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.591448 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz"] Jul 21 19:38:23.594050 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:23.594025 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac97b694_bcc5_47c3_a779_9bf9b79bbdf0.slice/crio-90661fa82908649a7617b85672fd186131ca3d161c8f766adf8660a4c6633e87 WatchSource:0}: Error finding container 90661fa82908649a7617b85672fd186131ca3d161c8f766adf8660a4c6633e87: Status 404 returned error can't find the container with id 90661fa82908649a7617b85672fd186131ca3d161c8f766adf8660a4c6633e87 Jul 21 19:38:23.675030 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.674957 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:23.675160 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:23.675064 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 19:38:23.675160 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:23.675115 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls podName:a341677b-760e-4573-b6f0-d90ef217c23e nodeName:}" failed. No retries permitted until 2026-07-21 19:38:24.675100254 +0000 UTC m=+67.859541915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mdjb5" (UID: "a341677b-760e-4573-b6f0-d90ef217c23e") : secret "samples-operator-tls" not found Jul 21 19:38:23.863192 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.863147 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" event={"ID":"d4272620-b668-47a1-b43d-ba5fd58231be","Type":"ContainerStarted","Data":"b9ce3dd077bc4dcd618093c7f5d20dcad6591389c90e89e76ba62f15ea6a51f2"} Jul 21 19:38:23.864407 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.864382 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kcl4p" event={"ID":"58826593-b4e6-4152-a0a9-a5b9746af2a9","Type":"ContainerStarted","Data":"93bff6db9c2533a94e52092446e685074a361ab8afe261b9a4bec8518fa475c3"} Jul 21 19:38:23.865583 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.865553 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" event={"ID":"639481df-2652-4b1d-b003-c6aef4eb44af","Type":"ContainerStarted","Data":"356cc6f3e49937688d8741b5d1cefb3ac2e6fce4a59e9cf8a51b9ced8444dd98"} Jul 21 19:38:23.866718 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.866698 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" event={"ID":"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0","Type":"ContainerStarted","Data":"90661fa82908649a7617b85672fd186131ca3d161c8f766adf8660a4c6633e87"} Jul 21 19:38:23.868003 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.867979 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kd25k" event={"ID":"aba01382-69a9-4443-b1a2-fe15605e698d","Type":"ContainerStarted","Data":"2b82e3eab84d502fb7e9129f1b33b5de1a999e1eea448af55493a1df61c14349"} Jul 21 19:38:23.869092 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:23.869072 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ll8xj" event={"ID":"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39","Type":"ContainerStarted","Data":"312578d4463bc2f7e6dd9e34bd2d2a3d4965046981b99cc5a8c921e0904c9d7e"} Jul 21 19:38:24.684752 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:24.684143 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:24.684752 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:24.684286 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 19:38:24.684752 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:24.684349 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls podName:a341677b-760e-4573-b6f0-d90ef217c23e nodeName:}" failed. No retries permitted until 2026-07-21 19:38:26.68432949 +0000 UTC m=+69.868771152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mdjb5" (UID: "a341677b-760e-4573-b6f0-d90ef217c23e") : secret "samples-operator-tls" not found Jul 21 19:38:25.490914 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:25.490673 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:38:25.490914 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:25.490918 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 19:38:25.491142 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:25.490985 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls podName:2c1b84bc-9f08-49b0-b538-c09d743953bf nodeName:}" failed. No retries permitted until 2026-07-21 19:38:57.490966051 +0000 UTC m=+100.675407710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls") pod "dns-default-dhqw9" (UID: "2c1b84bc-9f08-49b0-b538-c09d743953bf") : secret "dns-default-metrics-tls" not found Jul 21 19:38:26.698218 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:26.698180 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:26.698671 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:26.698343 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 19:38:26.698671 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:26.698411 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls podName:a341677b-760e-4573-b6f0-d90ef217c23e nodeName:}" failed. No retries permitted until 2026-07-21 19:38:30.698390719 +0000 UTC m=+73.882832386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mdjb5" (UID: "a341677b-760e-4573-b6f0-d90ef217c23e") : secret "samples-operator-tls" not found Jul 21 19:38:30.731912 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.731821 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:30.732447 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:30.732000 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 19:38:30.732447 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:30.732083 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls podName:a341677b-760e-4573-b6f0-d90ef217c23e nodeName:}" failed. No retries permitted until 2026-07-21 19:38:38.732061479 +0000 UTC m=+81.916503143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mdjb5" (UID: "a341677b-760e-4573-b6f0-d90ef217c23e") : secret "samples-operator-tls" not found Jul 21 19:38:30.892330 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.892296 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" event={"ID":"d4272620-b668-47a1-b43d-ba5fd58231be","Type":"ContainerStarted","Data":"f84bfd1c2bd0f2b2caa99cec9f56152d4c0d8f75c525d8c7020813ae66d240da"} Jul 21 19:38:30.894101 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.894062 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kcl4p" event={"ID":"58826593-b4e6-4152-a0a9-a5b9746af2a9","Type":"ContainerStarted","Data":"8c720a819be0318cdef9ed23bdaa202129b31acf31f2cda269c8aef177e95f59"} Jul 21 19:38:30.895942 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.895920 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/0.log" Jul 21 19:38:30.896059 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.895961 2557 generic.go:356] "Generic (PLEG): container finished" podID="639481df-2652-4b1d-b003-c6aef4eb44af" containerID="9c9448d964a9b4411b4d411eb5ad097b147551c6a2c369e186166a036928594e" exitCode=255 Jul 21 19:38:30.896059 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.896045 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" event={"ID":"639481df-2652-4b1d-b003-c6aef4eb44af","Type":"ContainerDied","Data":"9c9448d964a9b4411b4d411eb5ad097b147551c6a2c369e186166a036928594e"} Jul 21 19:38:30.896389 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.896213 2557 scope.go:117] "RemoveContainer" containerID="9c9448d964a9b4411b4d411eb5ad097b147551c6a2c369e186166a036928594e" Jul 21 19:38:30.897663 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.897618 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" event={"ID":"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0","Type":"ContainerStarted","Data":"f840cdd9cf010be1bb4df95bbfc08885088ff429fad30eb8bd47adb7559e2961"} Jul 21 19:38:30.899194 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.899161 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kd25k" event={"ID":"aba01382-69a9-4443-b1a2-fe15605e698d","Type":"ContainerStarted","Data":"487feeebe04019f9fc9d0a7d56e8db56065246e677bd4903210c6ccead341f88"} Jul 21 19:38:30.900845 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.900811 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ll8xj" event={"ID":"18203bb5-2bdc-42cb-ad2f-bfce2b2edd39","Type":"ContainerStarted","Data":"5e2857c465b3b0488941990973af60f48770f593236d4a869a3158c4cc1ca0f0"} Jul 21 19:38:30.901006 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.900993 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:38:30.908600 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.908554 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" podStartSLOduration=1.027216535 podStartE2EDuration="7.908537299s" totalImagesPullingTime="6.881320764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:38:30.907973152 +0000 UTC m=+74.092414860" watchObservedRunningTime="2026-07-21 19:38:30.908537299 +0000 UTC m=+74.092978976" Jul 21 19:38:30.946899 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.946821 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-kcl4p" podStartSLOduration=1.969550457 podStartE2EDuration="8.946801281s" totalImagesPullingTime="6.977250824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:38:30.92448041 +0000 UTC m=+74.108922110" watchObservedRunningTime="2026-07-21 19:38:30.946801281 +0000 UTC m=+74.131242965" Jul 21 19:38:30.962326 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.962279 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" podStartSLOduration=1.094217156 podStartE2EDuration="7.962259113s" totalImagesPullingTime="6.868041957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:38:30.96085511 +0000 UTC m=+74.145296823" watchObservedRunningTime="2026-07-21 19:38:30.962259113 +0000 UTC m=+74.146700795" Jul 21 19:38:30.978780 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:30.978734 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kd25k" podStartSLOduration=65.747263394 podStartE2EDuration="1m12.978715289s" totalImagesPullingTime="7.231451895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:38:30.978501815 +0000 UTC m=+74.162943497" watchObservedRunningTime="2026-07-21 19:38:30.978715289 +0000 UTC m=+74.163156976" Jul 21 19:38:31.552195 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.552124 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ll8xj" podStartSLOduration=66.586225578 podStartE2EDuration="1m13.552101443s" totalImagesPullingTime="6.965875865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:38:30.997674475 +0000 UTC m=+74.182116183" watchObservedRunningTime="2026-07-21 19:38:31.552101443 +0000 UTC m=+74.736543125" Jul 21 19:38:31.552397 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.552288 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mvk77"] Jul 21 19:38:31.555962 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.555936 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.564062 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.564040 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 19:38:31.564844 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.564815 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 19:38:31.573822 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.573800 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hqhr5\"" Jul 21 19:38:31.590277 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.590251 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mvk77"] Jul 21 19:38:31.646087 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.646056 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj9wf\" (UniqueName: \"kubernetes.io/projected/16431569-e917-4a3e-a020-0459e9a3c984-kube-api-access-zj9wf\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.646272 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.646105 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.646272 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.646188 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/16431569-e917-4a3e-a020-0459e9a3c984-data-volume\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.646272 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.646219 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.646272 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.646251 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/16431569-e917-4a3e-a020-0459e9a3c984-crio-socket\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.747514 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.747486 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/16431569-e917-4a3e-a020-0459e9a3c984-data-volume\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.747514 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.747517 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.748010 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.747539 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/16431569-e917-4a3e-a020-0459e9a3c984-crio-socket\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.748010 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.747610 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/16431569-e917-4a3e-a020-0459e9a3c984-crio-socket\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.748010 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:31.747620 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 19:38:31.748010 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:31.747696 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:32.24767613 +0000 UTC m=+75.432117796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : configmap "kube-rbac-proxy" not found Jul 21 19:38:31.748010 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.747718 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj9wf\" (UniqueName: \"kubernetes.io/projected/16431569-e917-4a3e-a020-0459e9a3c984-kube-api-access-zj9wf\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.748010 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.747762 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.748010 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.747864 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/16431569-e917-4a3e-a020-0459e9a3c984-data-volume\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.748010 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:31.747910 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 19:38:31.748010 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:31.747954 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:32.247940786 +0000 UTC m=+75.432382445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : secret "insights-runtime-extractor-tls" not found Jul 21 19:38:31.757125 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.757098 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj9wf\" (UniqueName: \"kubernetes.io/projected/16431569-e917-4a3e-a020-0459e9a3c984-kube-api-access-zj9wf\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:31.907069 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.907041 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:38:31.907496 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.907476 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/0.log" Jul 21 19:38:31.907588 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.907517 2557 generic.go:356] "Generic (PLEG): container finished" podID="639481df-2652-4b1d-b003-c6aef4eb44af" containerID="359e05bb9c21d632273ff3c1bfa091f3797b9ec918c5e05d2d08aab939d568ee" exitCode=255 Jul 21 19:38:31.907643 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.907618 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" event={"ID":"639481df-2652-4b1d-b003-c6aef4eb44af","Type":"ContainerDied","Data":"359e05bb9c21d632273ff3c1bfa091f3797b9ec918c5e05d2d08aab939d568ee"} Jul 21 19:38:31.907692 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.907679 2557 scope.go:117] "RemoveContainer" containerID="9c9448d964a9b4411b4d411eb5ad097b147551c6a2c369e186166a036928594e" Jul 21 19:38:31.907939 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:31.907919 2557 scope.go:117] "RemoveContainer" containerID="359e05bb9c21d632273ff3c1bfa091f3797b9ec918c5e05d2d08aab939d568ee" Jul 21 19:38:31.908261 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:31.908225 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-fvv9d_openshift-console-operator(639481df-2652-4b1d-b003-c6aef4eb44af)\"" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" podUID="639481df-2652-4b1d-b003-c6aef4eb44af" Jul 21 19:38:32.252519 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.252429 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:32.252676 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.252523 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:32.252676 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:32.252572 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 19:38:32.252676 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:32.252653 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 19:38:32.252808 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:32.252655 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:33.252634427 +0000 UTC m=+76.437076085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : configmap "kube-rbac-proxy" not found Jul 21 19:38:32.252808 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:32.252722 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:33.252710123 +0000 UTC m=+76.437151792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : secret "insights-runtime-extractor-tls" not found Jul 21 19:38:32.622633 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.622551 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7"] Jul 21 19:38:32.626546 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.626527 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:32.629666 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.629641 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 21 19:38:32.629851 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.629837 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 21 19:38:32.630008 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.629855 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6jc52\"" Jul 21 19:38:32.635201 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.635177 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7"] Jul 21 19:38:32.655922 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.655892 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:32.656031 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.655928 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:32.757024 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.756991 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:32.757370 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.757029 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:32.757370 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:32.757174 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 19:38:32.757370 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:32.757236 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert podName:3eb90b32-fc9a-4523-8348-cc6f777f6ac9 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:33.257220005 +0000 UTC m=+76.441661675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q4sp7" (UID: "3eb90b32-fc9a-4523-8348-cc6f777f6ac9") : secret "networking-console-plugin-cert" not found Jul 21 19:38:32.757661 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.757641 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:32.912747 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.912719 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:38:32.913118 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:32.913103 2557 scope.go:117] "RemoveContainer" containerID="359e05bb9c21d632273ff3c1bfa091f3797b9ec918c5e05d2d08aab939d568ee" Jul 21 19:38:32.913330 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:32.913308 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-fvv9d_openshift-console-operator(639481df-2652-4b1d-b003-c6aef4eb44af)\"" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" podUID="639481df-2652-4b1d-b003-c6aef4eb44af" Jul 21 19:38:33.216185 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:33.216104 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:33.216325 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:33.216261 2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:33.262185 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:33.262159 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:33.262260 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:33.262202 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:33.262260 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:33.262240 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:33.262329 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:33.262282 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 19:38:33.262329 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:33.262319 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 19:38:33.262400 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:33.262321 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 19:38:33.262400 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:33.262368 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:35.26234529 +0000 UTC m=+78.446786963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : configmap "kube-rbac-proxy" not found Jul 21 19:38:33.262400 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:33.262382 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:35.262376394 +0000 UTC m=+78.446818053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : secret "insights-runtime-extractor-tls" not found Jul 21 19:38:33.262400 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:33.262391 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert podName:3eb90b32-fc9a-4523-8348-cc6f777f6ac9 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:34.262386921 +0000 UTC m=+77.446828580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q4sp7" (UID: "3eb90b32-fc9a-4523-8348-cc6f777f6ac9") : secret "networking-console-plugin-cert" not found Jul 21 19:38:33.552204 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:33.552124 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-djwk4_02c2a5d8-4576-45b1-9c2a-ddbd5012322b/dns-node-resolver/0.log" Jul 21 19:38:33.916214 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:33.916185 2557 scope.go:117] "RemoveContainer" containerID="359e05bb9c21d632273ff3c1bfa091f3797b9ec918c5e05d2d08aab939d568ee" Jul 21 19:38:33.916642 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:33.916375 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-fvv9d_openshift-console-operator(639481df-2652-4b1d-b003-c6aef4eb44af)\"" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" podUID="639481df-2652-4b1d-b003-c6aef4eb44af" Jul 21 19:38:34.269744 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:34.269639 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:34.269940 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:34.269773 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 19:38:34.269940 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:34.269843 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert podName:3eb90b32-fc9a-4523-8348-cc6f777f6ac9 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:36.269824398 +0000 UTC m=+79.454266057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q4sp7" (UID: "3eb90b32-fc9a-4523-8348-cc6f777f6ac9") : secret "networking-console-plugin-cert" not found Jul 21 19:38:34.350258 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:34.350234 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-64pkq_0e3f4244-32a9-467b-b77c-26efe45e6aac/node-ca/0.log" Jul 21 19:38:34.918684 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:34.918654 2557 scope.go:117] "RemoveContainer" containerID="359e05bb9c21d632273ff3c1bfa091f3797b9ec918c5e05d2d08aab939d568ee" Jul 21 19:38:34.919095 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:34.918854 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-fvv9d_openshift-console-operator(639481df-2652-4b1d-b003-c6aef4eb44af)\"" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" podUID="639481df-2652-4b1d-b003-c6aef4eb44af" Jul 21 19:38:35.150076 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:35.150034 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-czf26_51ed889a-0a84-4523-8c9b-989d2ac767fe/serve-healthcheck-canary/0.log" Jul 21 19:38:35.278114 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:35.278031 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:35.278114 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:35.278104 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:35.278289 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:35.278168 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 19:38:35.278289 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:35.278193 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 19:38:35.278289 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:35.278246 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:39.278229248 +0000 UTC m=+82.462670912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : secret "insights-runtime-extractor-tls" not found Jul 21 19:38:35.278289 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:35.278260 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:39.278253918 +0000 UTC m=+82.462695577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : configmap "kube-rbac-proxy" not found Jul 21 19:38:36.285702 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:36.285667 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:36.286077 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:36.285824 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 19:38:36.286077 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:36.285910 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert podName:3eb90b32-fc9a-4523-8348-cc6f777f6ac9 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:40.28589272 +0000 UTC m=+83.470334391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q4sp7" (UID: "3eb90b32-fc9a-4523-8348-cc6f777f6ac9") : secret "networking-console-plugin-cert" not found Jul 21 19:38:38.806487 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:38.806443 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:38.808985 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:38.808958 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a341677b-760e-4573-b6f0-d90ef217c23e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mdjb5\" (UID: \"a341677b-760e-4573-b6f0-d90ef217c23e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:38.810693 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:38.810677 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" Jul 21 19:38:38.937659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:38.937634 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5"] Jul 21 19:38:39.310513 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:39.310406 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:39.310683 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:39.310524 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 19:38:39.310683 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:39.310586 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:47.310569904 +0000 UTC m=+90.495011567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : configmap "kube-rbac-proxy" not found Jul 21 19:38:39.310683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:39.310604 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:39.310818 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:39.310698 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 19:38:39.310818 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:39.310731 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:47.310720952 +0000 UTC m=+90.495162612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : secret "insights-runtime-extractor-tls" not found Jul 21 19:38:39.935957 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:39.935919 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" event={"ID":"a341677b-760e-4573-b6f0-d90ef217c23e","Type":"ContainerStarted","Data":"a039fea907f21d106d3fb1ba12fc25a3cb27cda3917c33404c27182dd74418f2"} Jul 21 19:38:40.319587 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:40.319494 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:40.319755 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:40.319671 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 19:38:40.319755 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:40.319748 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert podName:3eb90b32-fc9a-4523-8348-cc6f777f6ac9 nodeName:}" failed. No retries permitted until 2026-07-21 19:38:48.319728678 +0000 UTC m=+91.504170349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q4sp7" (UID: "3eb90b32-fc9a-4523-8348-cc6f777f6ac9") : secret "networking-console-plugin-cert" not found Jul 21 19:38:41.943264 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:41.943228 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" event={"ID":"a341677b-760e-4573-b6f0-d90ef217c23e","Type":"ContainerStarted","Data":"ded916d6662b5fd9ef1daee4a12898b58a347b3a90afeeb587f66950283aa271"} Jul 21 19:38:41.943745 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:41.943286 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" event={"ID":"a341677b-760e-4573-b6f0-d90ef217c23e","Type":"ContainerStarted","Data":"e0837a849ac736bf1a9900cf7ea70125f9f8d2ae190062254f78672af3621096"} Jul 21 19:38:41.960387 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:41.960344 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mdjb5" podStartSLOduration=17.995359937 podStartE2EDuration="19.960330165s" totalImagesPullingTime="1.964970228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:38:41.96010874 +0000 UTC m=+85.144550448" watchObservedRunningTime="2026-07-21 19:38:41.960330165 +0000 UTC m=+85.144771845" Jul 21 19:38:47.373890 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:47.373837 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:47.374265 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:47.373947 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:47.374265 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:47.374057 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 19:38:47.374265 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:38:47.374118 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls podName:16431569-e917-4a3e-a020-0459e9a3c984 nodeName:}" failed. No retries permitted until 2026-07-21 19:39:03.374104343 +0000 UTC m=+106.558546003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvk77" (UID: "16431569-e917-4a3e-a020-0459e9a3c984") : secret "insights-runtime-extractor-tls" not found Jul 21 19:38:47.374384 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:47.374338 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16431569-e917-4a3e-a020-0459e9a3c984-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:38:47.499049 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:47.499020 2557 scope.go:117] "RemoveContainer" containerID="359e05bb9c21d632273ff3c1bfa091f3797b9ec918c5e05d2d08aab939d568ee" Jul 21 19:38:47.966059 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:47.966033 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:38:47.966205 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:47.966093 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" event={"ID":"639481df-2652-4b1d-b003-c6aef4eb44af","Type":"ContainerStarted","Data":"8166a1a01ec192fb13532646c874b01655fda0caded083cd62d9924979f2ea3d"} Jul 21 19:38:47.966391 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:47.966369 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:47.983680 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:47.983626 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" podStartSLOduration=18.867497751 podStartE2EDuration="25.983613657s" totalImagesPullingTime="7.116115906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:38:47.982932678 +0000 UTC m=+91.167374381" watchObservedRunningTime="2026-07-21 19:38:47.983613657 +0000 UTC m=+91.168055369" Jul 21 19:38:48.088450 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:48.088423 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-fvv9d" Jul 21 19:38:48.383745 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:48.383659 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:48.386012 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:48.385993 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eb90b32-fc9a-4523-8348-cc6f777f6ac9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q4sp7\" (UID: \"3eb90b32-fc9a-4523-8348-cc6f777f6ac9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:48.535369 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:48.535335 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" Jul 21 19:38:48.659691 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:48.659670 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7"] Jul 21 19:38:48.661626 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:48.661601 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb90b32_fc9a_4523_8348_cc6f777f6ac9.slice/crio-0aa7c7c0b7801d04c0d4059fd1436521ac3f0b43b1ede4c67650eb9940d2e1b8 WatchSource:0}: Error finding container 0aa7c7c0b7801d04c0d4059fd1436521ac3f0b43b1ede4c67650eb9940d2e1b8: Status 404 returned error can't find the container with id 0aa7c7c0b7801d04c0d4059fd1436521ac3f0b43b1ede4c67650eb9940d2e1b8 Jul 21 19:38:48.970692 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:48.970602 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" event={"ID":"3eb90b32-fc9a-4523-8348-cc6f777f6ac9","Type":"ContainerStarted","Data":"0aa7c7c0b7801d04c0d4059fd1436521ac3f0b43b1ede4c67650eb9940d2e1b8"} Jul 21 19:38:49.975483 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:49.975440 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" event={"ID":"3eb90b32-fc9a-4523-8348-cc6f777f6ac9","Type":"ContainerStarted","Data":"14ab34af3648d85bdf2969da30b9a134cdda2cd8bacc8cebfaaa98634f23fc8d"} Jul 21 19:38:49.995084 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:49.995037 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q4sp7" podStartSLOduration=16.842769875 podStartE2EDuration="17.995020979s" totalImagesPullingTime="1.152251104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:38:49.994072377 +0000 UTC m=+93.178514080" watchObservedRunningTime="2026-07-21 19:38:49.995020979 +0000 UTC m=+93.179462659" Jul 21 19:38:57.178585 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.178552 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb"] Jul 21 19:38:57.182796 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.182774 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.186735 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.186717 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 19:38:57.187137 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.187113 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 19:38:57.187855 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.187835 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 19:38:57.188040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.188022 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 21 19:38:57.206931 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.206901 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb"] Jul 21 19:38:57.254992 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.254957 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbfg\" (UniqueName: \"kubernetes.io/projected/8358323e-9302-4238-b649-94c6a5207644-kube-api-access-tgbfg\") pod \"klusterlet-addon-workmgr-5b4bd9c668-gpzlb\" (UID: \"8358323e-9302-4238-b649-94c6a5207644\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.254992 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.254988 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8358323e-9302-4238-b649-94c6a5207644-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b4bd9c668-gpzlb\" (UID: \"8358323e-9302-4238-b649-94c6a5207644\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.255173 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.255020 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8358323e-9302-4238-b649-94c6a5207644-tmp\") pod \"klusterlet-addon-workmgr-5b4bd9c668-gpzlb\" (UID: \"8358323e-9302-4238-b649-94c6a5207644\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.356046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.356012 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8358323e-9302-4238-b649-94c6a5207644-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b4bd9c668-gpzlb\" (UID: \"8358323e-9302-4238-b649-94c6a5207644\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.356234 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.356069 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8358323e-9302-4238-b649-94c6a5207644-tmp\") pod \"klusterlet-addon-workmgr-5b4bd9c668-gpzlb\" (UID: \"8358323e-9302-4238-b649-94c6a5207644\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.356234 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.356137 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbfg\" (UniqueName: \"kubernetes.io/projected/8358323e-9302-4238-b649-94c6a5207644-kube-api-access-tgbfg\") pod \"klusterlet-addon-workmgr-5b4bd9c668-gpzlb\" (UID: \"8358323e-9302-4238-b649-94c6a5207644\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.356520 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.356500 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8358323e-9302-4238-b649-94c6a5207644-tmp\") pod \"klusterlet-addon-workmgr-5b4bd9c668-gpzlb\" (UID: \"8358323e-9302-4238-b649-94c6a5207644\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.358716 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.358692 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8358323e-9302-4238-b649-94c6a5207644-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b4bd9c668-gpzlb\" (UID: \"8358323e-9302-4238-b649-94c6a5207644\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.365133 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.365110 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7744fff668-gkfr6"] Jul 21 19:38:57.368218 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.368199 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.373068 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.373021 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 19:38:57.373068 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.373042 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 19:38:57.373821 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.373024 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 19:38:57.374534 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.374515 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7l5zt\"" Jul 21 19:38:57.375815 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.375794 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbfg\" (UniqueName: \"kubernetes.io/projected/8358323e-9302-4238-b649-94c6a5207644-kube-api-access-tgbfg\") pod \"klusterlet-addon-workmgr-5b4bd9c668-gpzlb\" (UID: \"8358323e-9302-4238-b649-94c6a5207644\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.380037 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.380014 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 19:38:57.396848 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.396813 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7744fff668-gkfr6"] Jul 21 19:38:57.456406 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.456380 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-ca-trust-extracted\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.456506 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.456427 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp455\" (UniqueName: \"kubernetes.io/projected/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-kube-api-access-dp455\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.456506 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.456446 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-bound-sa-token\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.456573 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.456512 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-image-registry-private-configuration\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.456621 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.456603 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-registry-tls\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.456657 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.456645 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-trusted-ca\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.456697 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.456685 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-registry-certificates\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.456739 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.456724 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-installation-pull-secrets\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.493630 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.493603 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:38:57.557996 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.557897 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-installation-pull-secrets\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.558132 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.558017 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-ca-trust-extracted\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.558132 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.558100 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp455\" (UniqueName: \"kubernetes.io/projected/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-kube-api-access-dp455\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.558237 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.558136 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-bound-sa-token\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.558237 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.558203 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-image-registry-private-configuration\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.558337 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.558242 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:38:57.558337 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.558284 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-registry-tls\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.558337 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.558310 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-trusted-ca\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.558462 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.558338 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-registry-certificates\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.558462 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.558410 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-ca-trust-extracted\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.559265 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.559221 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-registry-certificates\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.559596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.559570 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-trusted-ca\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.560303 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.560272 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-installation-pull-secrets\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.561186 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.561163 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1b84bc-9f08-49b0-b538-c09d743953bf-metrics-tls\") pod \"dns-default-dhqw9\" (UID: \"2c1b84bc-9f08-49b0-b538-c09d743953bf\") " pod="openshift-dns/dns-default-dhqw9" Jul 21 19:38:57.561318 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.561163 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-registry-tls\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.561439 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.561420 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-image-registry-private-configuration\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.568455 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.568433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp455\" (UniqueName: \"kubernetes.io/projected/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-kube-api-access-dp455\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.568950 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.568933 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87f3f5fc-bd83-40b6-b4cf-7a106e334aee-bound-sa-token\") pod \"image-registry-7744fff668-gkfr6\" (UID: \"87f3f5fc-bd83-40b6-b4cf-7a106e334aee\") " pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.594616 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.594585 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-46cqs\"" Jul 21 19:38:57.601123 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.601101 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dhqw9" Jul 21 19:38:57.628179 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.628143 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb"] Jul 21 19:38:57.630300 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:57.630274 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8358323e_9302_4238_b649_94c6a5207644.slice/crio-36f9db36f96999c3913f1cad8389a65c13885607ee388503cd01c60074597554 WatchSource:0}: Error finding container 36f9db36f96999c3913f1cad8389a65c13885607ee388503cd01c60074597554: Status 404 returned error can't find the container with id 36f9db36f96999c3913f1cad8389a65c13885607ee388503cd01c60074597554 Jul 21 19:38:57.682278 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.682225 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:57.741659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.741621 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dhqw9"] Jul 21 19:38:57.745171 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:57.745145 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1b84bc_9f08_49b0_b538_c09d743953bf.slice/crio-9a22adb4d98e09b995f2986a374c03dd7f120854343306c64d48053c54cdc33c WatchSource:0}: Error finding container 9a22adb4d98e09b995f2986a374c03dd7f120854343306c64d48053c54cdc33c: Status 404 returned error can't find the container with id 9a22adb4d98e09b995f2986a374c03dd7f120854343306c64d48053c54cdc33c Jul 21 19:38:57.828715 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:57.828679 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7744fff668-gkfr6"] Jul 21 19:38:57.831562 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:57.831534 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f3f5fc_bd83_40b6_b4cf_7a106e334aee.slice/crio-140798c3390fa1ce369fff0be1f7ea483af212beb637b96851da4f80c78f2800 WatchSource:0}: Error finding container 140798c3390fa1ce369fff0be1f7ea483af212beb637b96851da4f80c78f2800: Status 404 returned error can't find the container with id 140798c3390fa1ce369fff0be1f7ea483af212beb637b96851da4f80c78f2800 Jul 21 19:38:58.005094 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.005001 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dhqw9" event={"ID":"2c1b84bc-9f08-49b0-b538-c09d743953bf","Type":"ContainerStarted","Data":"9a22adb4d98e09b995f2986a374c03dd7f120854343306c64d48053c54cdc33c"} Jul 21 19:38:58.006535 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.006499 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7744fff668-gkfr6" event={"ID":"87f3f5fc-bd83-40b6-b4cf-7a106e334aee","Type":"ContainerStarted","Data":"dd1b7be928172daec0cc65c60f2037942bfd5eacf5b988050f5449a36cbf84e6"} Jul 21 19:38:58.006686 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.006554 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:38:58.006686 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.006577 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7744fff668-gkfr6" event={"ID":"87f3f5fc-bd83-40b6-b4cf-7a106e334aee","Type":"ContainerStarted","Data":"140798c3390fa1ce369fff0be1f7ea483af212beb637b96851da4f80c78f2800"} Jul 21 19:38:58.007668 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.007640 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" event={"ID":"8358323e-9302-4238-b649-94c6a5207644","Type":"ContainerStarted","Data":"36f9db36f96999c3913f1cad8389a65c13885607ee388503cd01c60074597554"} Jul 21 19:38:58.045815 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.045764 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7744fff668-gkfr6" podStartSLOduration=1.045748427 podStartE2EDuration="1.045748427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:38:58.045166991 +0000 UTC m=+101.229608698" watchObservedRunningTime="2026-07-21 19:38:58.045748427 +0000 UTC m=+101.230190109" Jul 21 19:38:58.121360 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.121320 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx"] Jul 21 19:38:58.124338 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.124319 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" Jul 21 19:38:58.127444 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.127414 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 19:38:58.127534 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.127424 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-b4q8j\"" Jul 21 19:38:58.143282 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.143251 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx"] Jul 21 19:38:58.264281 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.264193 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d14e31a3-1f62-489d-9d7a-32bb322e155c-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-kqpcx\" (UID: \"d14e31a3-1f62-489d-9d7a-32bb322e155c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" Jul 21 19:38:58.364791 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.364761 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d14e31a3-1f62-489d-9d7a-32bb322e155c-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-kqpcx\" (UID: \"d14e31a3-1f62-489d-9d7a-32bb322e155c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" Jul 21 19:38:58.367685 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.367654 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d14e31a3-1f62-489d-9d7a-32bb322e155c-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-kqpcx\" (UID: \"d14e31a3-1f62-489d-9d7a-32bb322e155c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" Jul 21 19:38:58.433495 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.433459 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" Jul 21 19:38:58.602789 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:58.602750 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx"] Jul 21 19:38:58.605381 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:38:58.605343 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14e31a3_1f62_489d_9d7a_32bb322e155c.slice/crio-9eda3146d2700f8fcb990302fe80bfdb21ab5a9b82c4c91b82126f305c24bae1 WatchSource:0}: Error finding container 9eda3146d2700f8fcb990302fe80bfdb21ab5a9b82c4c91b82126f305c24bae1: Status 404 returned error can't find the container with id 9eda3146d2700f8fcb990302fe80bfdb21ab5a9b82c4c91b82126f305c24bae1 Jul 21 19:38:59.014304 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:38:59.014267 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" event={"ID":"d14e31a3-1f62-489d-9d7a-32bb322e155c","Type":"ContainerStarted","Data":"9eda3146d2700f8fcb990302fe80bfdb21ab5a9b82c4c91b82126f305c24bae1"} Jul 21 19:39:01.910941 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:01.910910 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ll8xj" Jul 21 19:39:03.030315 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.030281 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dhqw9" event={"ID":"2c1b84bc-9f08-49b0-b538-c09d743953bf","Type":"ContainerStarted","Data":"dc0b24ad7967074d4718ec86631bb93ef473c672c9a06cde03086e7d24dab0f8"} Jul 21 19:39:03.031692 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.031668 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" event={"ID":"d14e31a3-1f62-489d-9d7a-32bb322e155c","Type":"ContainerStarted","Data":"8fcf2b65554b7c25957d847606f80ff7eeaced890572fd556b0ea841b2c0685f"} Jul 21 19:39:03.031821 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.031711 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" Jul 21 19:39:03.033294 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.033135 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" event={"ID":"8358323e-9302-4238-b649-94c6a5207644","Type":"ContainerStarted","Data":"037b13485a26d31568697b47293398faa0c4b8938da1e9bcd9335243c1bdc5f8"} Jul 21 19:39:03.033653 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.033608 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:39:03.034772 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.034752 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" Jul 21 19:39:03.036811 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.036793 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" Jul 21 19:39:03.051349 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.051311 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kqpcx" podStartSLOduration=0.883946615 podStartE2EDuration="5.051300823s" totalImagesPullingTime="4.167354208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:03.050089751 +0000 UTC m=+106.234531453" watchObservedRunningTime="2026-07-21 19:39:03.051300823 +0000 UTC m=+106.235742504" Jul 21 19:39:03.066826 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.066788 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b4bd9c668-gpzlb" podStartSLOduration=0.906828219 podStartE2EDuration="6.066775652s" totalImagesPullingTime="5.159947433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:03.066128167 +0000 UTC m=+106.250569869" watchObservedRunningTime="2026-07-21 19:39:03.066775652 +0000 UTC m=+106.251217334" Jul 21 19:39:03.378160 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.378077 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75c9ddb8c4-m52qg"] Jul 21 19:39:03.398967 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.398942 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c9ddb8c4-m52qg"] Jul 21 19:39:03.399105 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.399040 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.403318 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.403294 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 19:39:03.403535 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.403515 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 19:39:03.403656 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.403566 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 19:39:03.403656 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.403640 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 19:39:03.403756 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.403570 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 19:39:03.404256 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.404241 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 19:39:03.404539 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.404514 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vr2bh\"" Jul 21 19:39:03.404645 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.404539 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 19:39:03.408155 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.408136 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 19:39:03.409079 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.409062 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:39:03.411237 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.411214 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16431569-e917-4a3e-a020-0459e9a3c984-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvk77\" (UID: \"16431569-e917-4a3e-a020-0459e9a3c984\") " pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:39:03.510083 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.510050 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-oauth-serving-cert\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.510244 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.510096 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-trusted-ca-bundle\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.510244 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.510144 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-oauth-config\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.510244 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.510183 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxzjv\" (UniqueName: \"kubernetes.io/projected/5c919420-2785-4a17-a4ce-8728aa2ad953-kube-api-access-lxzjv\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.510244 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.510223 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-serving-cert\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.510244 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.510243 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-console-config\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.510401 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.510268 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-service-ca\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.610744 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.610712 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-oauth-serving-cert\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.610957 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.610756 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-trusted-ca-bundle\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.610957 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.610781 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-oauth-config\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.610957 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.610799 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzjv\" (UniqueName: \"kubernetes.io/projected/5c919420-2785-4a17-a4ce-8728aa2ad953-kube-api-access-lxzjv\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.610957 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.610815 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-serving-cert\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.610957 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.610833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-console-config\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.610957 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.610864 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-service-ca\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.611626 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.611603 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-service-ca\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.611626 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.611612 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-console-config\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.611779 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.611603 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-oauth-serving-cert\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.611894 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.611856 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-trusted-ca-bundle\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.613332 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.613312 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-serving-cert\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.613411 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.613367 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-oauth-config\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.619416 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.619399 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxzjv\" (UniqueName: \"kubernetes.io/projected/5c919420-2785-4a17-a4ce-8728aa2ad953-kube-api-access-lxzjv\") pod \"console-75c9ddb8c4-m52qg\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.666420 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.666401 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mvk77" Jul 21 19:39:03.709149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.709108 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:03.852703 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.852667 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mvk77"] Jul 21 19:39:03.860891 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:39:03.860844 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16431569_e917_4a3e_a020_0459e9a3c984.slice/crio-c14ed4d7262e996d2b68c073b97f03fdb3fb986be2272a7617bb1fae2e201231 WatchSource:0}: Error finding container c14ed4d7262e996d2b68c073b97f03fdb3fb986be2272a7617bb1fae2e201231: Status 404 returned error can't find the container with id c14ed4d7262e996d2b68c073b97f03fdb3fb986be2272a7617bb1fae2e201231 Jul 21 19:39:03.889908 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:03.889869 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c9ddb8c4-m52qg"] Jul 21 19:39:03.891816 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:39:03.891793 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c919420_2785_4a17_a4ce_8728aa2ad953.slice/crio-593eddedf5b73b355d0b7e5bddd70c37135b34fd105b0ab68c40f8ceb40a8c4b WatchSource:0}: Error finding container 593eddedf5b73b355d0b7e5bddd70c37135b34fd105b0ab68c40f8ceb40a8c4b: Status 404 returned error can't find the container with id 593eddedf5b73b355d0b7e5bddd70c37135b34fd105b0ab68c40f8ceb40a8c4b Jul 21 19:39:04.038089 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.037993 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dhqw9" event={"ID":"2c1b84bc-9f08-49b0-b538-c09d743953bf","Type":"ContainerStarted","Data":"780605b05d457215f2d67dd2c0deb9fa46c3bfcf18f30d782e15dfcdaa73f065"} Jul 21 19:39:04.038517 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.038101 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dhqw9" Jul 21 19:39:04.039284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.039263 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c9ddb8c4-m52qg" event={"ID":"5c919420-2785-4a17-a4ce-8728aa2ad953","Type":"ContainerStarted","Data":"593eddedf5b73b355d0b7e5bddd70c37135b34fd105b0ab68c40f8ceb40a8c4b"} Jul 21 19:39:04.040324 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.040304 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvk77" event={"ID":"16431569-e917-4a3e-a020-0459e9a3c984","Type":"ContainerStarted","Data":"c14ed4d7262e996d2b68c073b97f03fdb3fb986be2272a7617bb1fae2e201231"} Jul 21 19:39:04.058481 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.058440 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dhqw9" podStartSLOduration=66.02936863 podStartE2EDuration="1m11.05842737s" totalImagesPullingTime="5.02905874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:04.057691804 +0000 UTC m=+107.242133485" watchObservedRunningTime="2026-07-21 19:39:04.05842737 +0000 UTC m=+107.242869062" Jul 21 19:39:04.166019 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.165992 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-98k99"] Jul 21 19:39:04.200684 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.200651 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-98k99"] Jul 21 19:39:04.200684 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.200684 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.204043 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.204018 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 19:39:04.204170 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.204053 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 19:39:04.204170 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.204062 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 19:39:04.204170 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.204082 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 19:39:04.204170 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.204089 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 19:39:04.204170 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.204055 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cmgvg\"" Jul 21 19:39:04.317610 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.317533 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eef2534-1587-4685-9062-a783e3b7ed3e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.317747 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.317624 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7796x\" (UniqueName: \"kubernetes.io/projected/0eef2534-1587-4685-9062-a783e3b7ed3e-kube-api-access-7796x\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.317747 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.317676 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0eef2534-1587-4685-9062-a783e3b7ed3e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.317747 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.317696 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0eef2534-1587-4685-9062-a783e3b7ed3e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.418528 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.418485 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eef2534-1587-4685-9062-a783e3b7ed3e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.418711 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.418548 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7796x\" (UniqueName: \"kubernetes.io/projected/0eef2534-1587-4685-9062-a783e3b7ed3e-kube-api-access-7796x\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.418711 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.418587 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0eef2534-1587-4685-9062-a783e3b7ed3e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.418711 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.418617 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0eef2534-1587-4685-9062-a783e3b7ed3e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.419406 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.419373 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eef2534-1587-4685-9062-a783e3b7ed3e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.425615 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.425589 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0eef2534-1587-4685-9062-a783e3b7ed3e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.426527 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.426500 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0eef2534-1587-4685-9062-a783e3b7ed3e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.430386 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.430356 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7796x\" (UniqueName: \"kubernetes.io/projected/0eef2534-1587-4685-9062-a783e3b7ed3e-kube-api-access-7796x\") pod \"prometheus-operator-5945b5cfd-98k99\" (UID: \"0eef2534-1587-4685-9062-a783e3b7ed3e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.510431 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.510404 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" Jul 21 19:39:04.684718 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:04.684687 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-98k99"] Jul 21 19:39:04.687725 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:39:04.687676 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eef2534_1587_4685_9062_a783e3b7ed3e.slice/crio-e6bafb4ce9b63db6f54947681dc71b26c8b4cfc6bf382efb0189e8443cd543aa WatchSource:0}: Error finding container e6bafb4ce9b63db6f54947681dc71b26c8b4cfc6bf382efb0189e8443cd543aa: Status 404 returned error can't find the container with id e6bafb4ce9b63db6f54947681dc71b26c8b4cfc6bf382efb0189e8443cd543aa Jul 21 19:39:05.046868 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:05.046691 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" event={"ID":"0eef2534-1587-4685-9062-a783e3b7ed3e","Type":"ContainerStarted","Data":"e6bafb4ce9b63db6f54947681dc71b26c8b4cfc6bf382efb0189e8443cd543aa"} Jul 21 19:39:11.071652 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:11.071403 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c9ddb8c4-m52qg" event={"ID":"5c919420-2785-4a17-a4ce-8728aa2ad953","Type":"ContainerStarted","Data":"f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6"} Jul 21 19:39:11.073466 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:11.073433 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvk77" event={"ID":"16431569-e917-4a3e-a020-0459e9a3c984","Type":"ContainerStarted","Data":"09ccc850dc45c92b33c593f6512c4c62697266684c7f6e32688423718f2d799e"} Jul 21 19:39:11.075850 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:11.075768 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" event={"ID":"0eef2534-1587-4685-9062-a783e3b7ed3e","Type":"ContainerStarted","Data":"623655fcb08d3ae38a2e1462c466e7ff3296373492b457741679754328ec92d8"} Jul 21 19:39:11.075850 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:11.075820 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" event={"ID":"0eef2534-1587-4685-9062-a783e3b7ed3e","Type":"ContainerStarted","Data":"da32c805e15761ebe975deda8d068f9e4cc8854593dd865baa6ec2203e97fb8a"} Jul 21 19:39:11.091825 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:11.091779 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75c9ddb8c4-m52qg" podStartSLOduration=1.86010853 podStartE2EDuration="8.091760071s" totalImagesPullingTime="6.231651541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:11.09005947 +0000 UTC m=+114.274501177" watchObservedRunningTime="2026-07-21 19:39:11.091760071 +0000 UTC m=+114.276201759" Jul 21 19:39:11.108366 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:11.108309 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-98k99" podStartSLOduration=1.6731750170000002 podStartE2EDuration="7.108291621s" totalImagesPullingTime="5.435116604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:11.108028275 +0000 UTC m=+114.292469982" watchObservedRunningTime="2026-07-21 19:39:11.108291621 +0000 UTC m=+114.292733304" Jul 21 19:39:12.538989 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.538951 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f4f4w"] Jul 21 19:39:12.557822 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.557790 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.561103 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.561081 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 19:39:12.561447 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.561422 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6t7pw\"" Jul 21 19:39:12.561563 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.561456 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 19:39:12.561563 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.561425 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 19:39:12.703217 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.703178 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.703399 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.703284 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vnn\" (UniqueName: \"kubernetes.io/projected/e2ee63bc-ee4d-4646-b833-b57fe5467935-kube-api-access-d9vnn\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.703399 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.703333 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2ee63bc-ee4d-4646-b833-b57fe5467935-root\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.703399 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.703376 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-tls\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.703529 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.703442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-wtmp\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.703529 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.703508 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2ee63bc-ee4d-4646-b833-b57fe5467935-sys\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.703606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.703536 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-textfile\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.703606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.703567 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-accelerators-collector-config\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.703698 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.703640 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2ee63bc-ee4d-4646-b833-b57fe5467935-metrics-client-ca\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804203 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804111 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2ee63bc-ee4d-4646-b833-b57fe5467935-sys\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804203 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804159 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-textfile\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804203 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804197 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-accelerators-collector-config\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804491 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804222 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2ee63bc-ee4d-4646-b833-b57fe5467935-sys\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804491 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804226 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2ee63bc-ee4d-4646-b833-b57fe5467935-metrics-client-ca\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804491 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804491 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804346 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vnn\" (UniqueName: \"kubernetes.io/projected/e2ee63bc-ee4d-4646-b833-b57fe5467935-kube-api-access-d9vnn\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804491 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804376 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2ee63bc-ee4d-4646-b833-b57fe5467935-root\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804491 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804409 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-tls\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804491 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804462 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-wtmp\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804830 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804618 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-wtmp\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804830 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804814 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-textfile\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804955 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804822 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2ee63bc-ee4d-4646-b833-b57fe5467935-metrics-client-ca\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.804955 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.804886 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2ee63bc-ee4d-4646-b833-b57fe5467935-root\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.805036 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:39:12.804957 2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 19:39:12.805036 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:39:12.805012 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-tls podName:e2ee63bc-ee4d-4646-b833-b57fe5467935 nodeName:}" failed. No retries permitted until 2026-07-21 19:39:13.304996009 +0000 UTC m=+116.489437671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-tls") pod "node-exporter-f4f4w" (UID: "e2ee63bc-ee4d-4646-b833-b57fe5467935") : secret "node-exporter-tls" not found Jul 21 19:39:12.805267 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.805242 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-accelerators-collector-config\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.807095 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.807071 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:12.819493 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:12.819470 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vnn\" (UniqueName: \"kubernetes.io/projected/e2ee63bc-ee4d-4646-b833-b57fe5467935-kube-api-access-d9vnn\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:13.309057 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:13.309023 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-tls\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:13.311996 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:13.311959 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2ee63bc-ee4d-4646-b833-b57fe5467935-node-exporter-tls\") pod \"node-exporter-f4f4w\" (UID: \"e2ee63bc-ee4d-4646-b833-b57fe5467935\") " pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:13.468193 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:13.468137 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f4f4w" Jul 21 19:39:13.611151 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:39:13.611121 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ee63bc_ee4d_4646_b833_b57fe5467935.slice/crio-4a7a74bfbf33993c057de4169b000b4549c1a4193be86881d2d94e3148b77ca7 WatchSource:0}: Error finding container 4a7a74bfbf33993c057de4169b000b4549c1a4193be86881d2d94e3148b77ca7: Status 404 returned error can't find the container with id 4a7a74bfbf33993c057de4169b000b4549c1a4193be86881d2d94e3148b77ca7 Jul 21 19:39:13.709518 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:13.709288 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:13.709651 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:13.709577 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:13.714477 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:13.714449 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:14.048821 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:14.048792 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dhqw9" Jul 21 19:39:14.093050 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:14.093012 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvk77" event={"ID":"16431569-e917-4a3e-a020-0459e9a3c984","Type":"ContainerStarted","Data":"9c0dac8d2dde8d1e713b42131161ddb1b626fe173f669466568efe6114d6f6a1"} Jul 21 19:39:14.094689 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:14.094657 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f4f4w" event={"ID":"e2ee63bc-ee4d-4646-b833-b57fe5467935","Type":"ContainerStarted","Data":"4a7a74bfbf33993c057de4169b000b4549c1a4193be86881d2d94e3148b77ca7"} Jul 21 19:39:14.099688 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:14.099487 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:39:15.100847 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:15.100810 2557 generic.go:356] "Generic (PLEG): container finished" podID="e2ee63bc-ee4d-4646-b833-b57fe5467935" containerID="8bf7bc4f5bc6d4d8a3a834fb26cec881177b86989aca2f757e959c15ae0dd183" exitCode=0 Jul 21 19:39:15.101346 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:15.100915 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f4f4w" event={"ID":"e2ee63bc-ee4d-4646-b833-b57fe5467935","Type":"ContainerDied","Data":"8bf7bc4f5bc6d4d8a3a834fb26cec881177b86989aca2f757e959c15ae0dd183"} Jul 21 19:39:16.107692 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.107653 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f4f4w" event={"ID":"e2ee63bc-ee4d-4646-b833-b57fe5467935","Type":"ContainerStarted","Data":"703141a8b1379df554b47c78db8f9545ebac226fe7fceb08953c6d25913ad2fe"} Jul 21 19:39:16.108171 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.107712 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f4f4w" event={"ID":"e2ee63bc-ee4d-4646-b833-b57fe5467935","Type":"ContainerStarted","Data":"4c8520b1967ef7c01378ebecd1df7e8307f181df3fcd53db0fc04aad6eabf1c2"} Jul 21 19:39:16.127548 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.127497 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f4f4w" podStartSLOduration=3.079581812 podStartE2EDuration="4.127479697s" totalImagesPullingTime="973.588245ms" totalInitContainerRuntime="74.30964ms" isStatefulPod=true podCreationTimestamp="2026-07-21 19:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:16.125931227 +0000 UTC m=+119.310372934" watchObservedRunningTime="2026-07-21 19:39:16.127479697 +0000 UTC m=+119.311921418" Jul 21 19:39:16.837844 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.836089 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-58ffcbfbc7-6hw54"] Jul 21 19:39:16.861379 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.861337 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58ffcbfbc7-6hw54"] Jul 21 19:39:16.861553 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.861492 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:16.864331 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.864299 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 19:39:16.865770 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.865552 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 19:39:16.865770 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.865624 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 19:39:16.865770 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.865645 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 19:39:16.865770 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.865633 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eui9opl6vk0jq\"" Jul 21 19:39:16.866626 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:16.866457 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rtn42\"" Jul 21 19:39:17.045595 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.045558 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-secret-metrics-server-tls\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.045778 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.045607 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-client-ca-bundle\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.045778 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.045703 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7gmp\" (UniqueName: \"kubernetes.io/projected/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-kube-api-access-s7gmp\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.045913 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.045783 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-audit-log\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.045913 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.045825 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.045913 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.045853 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-secret-metrics-server-client-certs\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.045913 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.045903 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-metrics-server-audit-profiles\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.146648 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.146617 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7gmp\" (UniqueName: \"kubernetes.io/projected/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-kube-api-access-s7gmp\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.147131 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.146657 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-audit-log\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.147131 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.146683 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.147131 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.146712 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-secret-metrics-server-client-certs\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.147131 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.146728 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-metrics-server-audit-profiles\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.147131 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.146836 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-secret-metrics-server-tls\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.147131 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.146865 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-client-ca-bundle\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.147359 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.147119 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-audit-log\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.149541 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.149515 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 19:39:17.149686 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.149515 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 19:39:17.149686 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.149605 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eui9opl6vk0jq\"" Jul 21 19:39:17.149686 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.149623 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 19:39:17.149686 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.149664 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 19:39:17.155324 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.155303 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7gmp\" (UniqueName: \"kubernetes.io/projected/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-kube-api-access-s7gmp\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.157716 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.157676 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.157828 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.157803 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-metrics-server-audit-profiles\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.159710 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.159690 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-secret-metrics-server-tls\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.159894 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.159859 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-client-ca-bundle\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.159973 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.159865 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a77c31f0-8c2f-4a2e-b716-bfa10cd731e1-secret-metrics-server-client-certs\") pod \"metrics-server-58ffcbfbc7-6hw54\" (UID: \"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1\") " pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.176273 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.176244 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rtn42\"" Jul 21 19:39:17.184106 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.184072 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:17.385831 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:17.385802 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58ffcbfbc7-6hw54"] Jul 21 19:39:17.388650 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:39:17.388627 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77c31f0_8c2f_4a2e_b716_bfa10cd731e1.slice/crio-d1a42cbac0fcd21e059acae1b02c52d201b1a10f6267ed12a3365e5213c43750 WatchSource:0}: Error finding container d1a42cbac0fcd21e059acae1b02c52d201b1a10f6267ed12a3365e5213c43750: Status 404 returned error can't find the container with id d1a42cbac0fcd21e059acae1b02c52d201b1a10f6267ed12a3365e5213c43750 Jul 21 19:39:18.117162 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:18.117127 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvk77" event={"ID":"16431569-e917-4a3e-a020-0459e9a3c984","Type":"ContainerStarted","Data":"08afaaad7ff6565be21215afd5cb124fb477d018c2b140c0f5133a8d73ed35bd"} Jul 21 19:39:18.118365 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:18.118340 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" event={"ID":"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1","Type":"ContainerStarted","Data":"d1a42cbac0fcd21e059acae1b02c52d201b1a10f6267ed12a3365e5213c43750"} Jul 21 19:39:18.135445 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:18.135401 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mvk77" podStartSLOduration=33.871366205 podStartE2EDuration="47.135387533s" totalImagesPullingTime="13.264021328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 19:38:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:18.134952118 +0000 UTC m=+121.319393826" watchObservedRunningTime="2026-07-21 19:39:18.135387533 +0000 UTC m=+121.319829213" Jul 21 19:39:19.021203 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:19.021170 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7744fff668-gkfr6" Jul 21 19:39:20.130397 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:20.130357 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" event={"ID":"a77c31f0-8c2f-4a2e-b716-bfa10cd731e1","Type":"ContainerStarted","Data":"1ba169ae33a01bff3d975d3e6254149248ae30553d34bbabe77ea7cab070fa01"} Jul 21 19:39:20.148772 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:20.148723 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" podStartSLOduration=2.236823258 podStartE2EDuration="4.148707947s" totalImagesPullingTime="1.911884689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:20.148650667 +0000 UTC m=+123.333092399" watchObservedRunningTime="2026-07-21 19:39:20.148707947 +0000 UTC m=+123.333149627" Jul 21 19:39:22.176151 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.176118 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-96ccc48f6-nwb2r"] Jul 21 19:39:22.181830 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.181810 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.185702 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.185680 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-trusted-ca-bundle\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.185803 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.185711 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5stj\" (UniqueName: \"kubernetes.io/projected/23d4a22e-1d5e-46ac-be0a-a473d93c102e-kube-api-access-d5stj\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.185803 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.185760 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-oauth-config\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.185926 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.185814 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-serving-cert\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.185926 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.185849 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-oauth-serving-cert\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.186005 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.185942 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-service-ca\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.186005 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.185961 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-config\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.195533 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.191379 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-96ccc48f6-nwb2r"] Jul 21 19:39:22.287167 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.287130 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-service-ca\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.287167 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.287166 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-config\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.287410 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.287186 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-trusted-ca-bundle\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.287410 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.287326 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5stj\" (UniqueName: \"kubernetes.io/projected/23d4a22e-1d5e-46ac-be0a-a473d93c102e-kube-api-access-d5stj\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.287410 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.287391 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-oauth-config\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.287555 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.287433 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-serving-cert\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.287555 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.287457 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-oauth-serving-cert\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.287951 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.287919 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-service-ca\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.288065 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.287925 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-config\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.288143 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.288123 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-trusted-ca-bundle\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.288181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.288161 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-oauth-serving-cert\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.290015 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.289993 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-serving-cert\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.290108 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.290019 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-oauth-config\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.295538 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.295520 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5stj\" (UniqueName: \"kubernetes.io/projected/23d4a22e-1d5e-46ac-be0a-a473d93c102e-kube-api-access-d5stj\") pod \"console-96ccc48f6-nwb2r\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.491404 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.491319 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:22.618547 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:22.618416 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-96ccc48f6-nwb2r"] Jul 21 19:39:22.621257 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:39:22.621225 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d4a22e_1d5e_46ac_be0a_a473d93c102e.slice/crio-b52b230d9ade28aa9d61c355db1abbb828e40c7b4d1f7c58e7a6fc3cf11ed362 WatchSource:0}: Error finding container b52b230d9ade28aa9d61c355db1abbb828e40c7b4d1f7c58e7a6fc3cf11ed362: Status 404 returned error can't find the container with id b52b230d9ade28aa9d61c355db1abbb828e40c7b4d1f7c58e7a6fc3cf11ed362 Jul 21 19:39:23.142992 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:23.142958 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-96ccc48f6-nwb2r" event={"ID":"23d4a22e-1d5e-46ac-be0a-a473d93c102e","Type":"ContainerStarted","Data":"11a602f294b181ade2cfc63a46507446cdd4d2ff0f163129603976463c755544"} Jul 21 19:39:23.143179 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:23.143014 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-96ccc48f6-nwb2r" event={"ID":"23d4a22e-1d5e-46ac-be0a-a473d93c102e","Type":"ContainerStarted","Data":"b52b230d9ade28aa9d61c355db1abbb828e40c7b4d1f7c58e7a6fc3cf11ed362"} Jul 21 19:39:23.165347 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:23.165297 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-96ccc48f6-nwb2r" podStartSLOduration=1.165281335 podStartE2EDuration="1.165281335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:23.163043384 +0000 UTC m=+126.347485086" watchObservedRunningTime="2026-07-21 19:39:23.165281335 +0000 UTC m=+126.349723015" Jul 21 19:39:26.001113 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.001081 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-96ccc48f6-nwb2r"] Jul 21 19:39:26.035472 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.035442 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-785f8fdb44-t94ff"] Jul 21 19:39:26.039149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.039124 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.049185 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.049141 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-785f8fdb44-t94ff"] Jul 21 19:39:26.118691 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.118652 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-config\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.118856 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.118706 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwct\" (UniqueName: \"kubernetes.io/projected/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-kube-api-access-2zwct\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.118856 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.118734 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-oauth-serving-cert\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.118856 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.118770 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-oauth-config\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.118856 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.118820 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-serving-cert\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.119017 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.118908 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-service-ca\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.119017 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.118978 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-trusted-ca-bundle\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.219361 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.219319 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-service-ca\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.219534 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.219384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-trusted-ca-bundle\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.219534 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.219431 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-config\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.219534 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.219454 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwct\" (UniqueName: \"kubernetes.io/projected/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-kube-api-access-2zwct\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.219534 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.219472 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-oauth-serving-cert\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.219534 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.219499 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-oauth-config\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.219534 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.219526 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-serving-cert\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.220152 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.220122 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-service-ca\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.220255 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.220212 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-config\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.220391 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.220373 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-oauth-serving-cert\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.220443 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.220424 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-trusted-ca-bundle\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.222479 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.222451 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-oauth-config\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.222671 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.222649 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-serving-cert\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.227357 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.227337 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwct\" (UniqueName: \"kubernetes.io/projected/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-kube-api-access-2zwct\") pod \"console-785f8fdb44-t94ff\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.349695 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.349602 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:26.475546 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:26.475509 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-785f8fdb44-t94ff"] Jul 21 19:39:26.478286 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:39:26.478249 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793d3b3b_c45e_4397_bf0d_cddb1f665f7d.slice/crio-810efe870cd463fe4a915077745cd3269381019ef1849cad3697ead4b286d93c WatchSource:0}: Error finding container 810efe870cd463fe4a915077745cd3269381019ef1849cad3697ead4b286d93c: Status 404 returned error can't find the container with id 810efe870cd463fe4a915077745cd3269381019ef1849cad3697ead4b286d93c Jul 21 19:39:27.126941 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:27.126902 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:39:27.129148 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:27.129129 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30e72480-263d-4449-87d6-4a30fdbcfcee-metrics-certs\") pod \"network-metrics-daemon-fjfkl\" (UID: \"30e72480-263d-4449-87d6-4a30fdbcfcee\") " pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:39:27.163677 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:27.163641 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785f8fdb44-t94ff" event={"ID":"793d3b3b-c45e-4397-bf0d-cddb1f665f7d","Type":"ContainerStarted","Data":"607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a"} Jul 21 19:39:27.163830 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:27.163697 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785f8fdb44-t94ff" event={"ID":"793d3b3b-c45e-4397-bf0d-cddb1f665f7d","Type":"ContainerStarted","Data":"810efe870cd463fe4a915077745cd3269381019ef1849cad3697ead4b286d93c"} Jul 21 19:39:27.182270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:27.182218 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-785f8fdb44-t94ff" podStartSLOduration=1.182201777 podStartE2EDuration="1.182201777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:27.181884733 +0000 UTC m=+130.366326433" watchObservedRunningTime="2026-07-21 19:39:27.182201777 +0000 UTC m=+130.366643508" Jul 21 19:39:27.317422 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:27.317394 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tcjpf\"" Jul 21 19:39:27.325040 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:27.325014 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjfkl" Jul 21 19:39:27.457194 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:27.457167 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fjfkl"] Jul 21 19:39:27.458506 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:39:27.458481 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e72480_263d_4449_87d6_4a30fdbcfcee.slice/crio-d55fd569b5750dadc1fc06c91af4aeb843839be9cd8948fe06f91b1756d2d193 WatchSource:0}: Error finding container d55fd569b5750dadc1fc06c91af4aeb843839be9cd8948fe06f91b1756d2d193: Status 404 returned error can't find the container with id d55fd569b5750dadc1fc06c91af4aeb843839be9cd8948fe06f91b1756d2d193 Jul 21 19:39:28.169596 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:28.169552 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fjfkl" event={"ID":"30e72480-263d-4449-87d6-4a30fdbcfcee","Type":"ContainerStarted","Data":"d55fd569b5750dadc1fc06c91af4aeb843839be9cd8948fe06f91b1756d2d193"} Jul 21 19:39:29.175011 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:29.174955 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fjfkl" event={"ID":"30e72480-263d-4449-87d6-4a30fdbcfcee","Type":"ContainerStarted","Data":"ded3867fbe216a96e7161cbaa68389497da1ec097f062f200b1394a1d09e00d2"} Jul 21 19:39:29.175364 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:29.175061 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fjfkl" event={"ID":"30e72480-263d-4449-87d6-4a30fdbcfcee","Type":"ContainerStarted","Data":"5119ab353e8530c3c960a4eebd18fa2c923a77d124497d57dd4330085c9b502d"} Jul 21 19:39:29.192296 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:29.192245 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fjfkl" podStartSLOduration=129.988208893 podStartE2EDuration="2m11.19223074s" totalImagesPullingTime="1.204021847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:39:29.192201692 +0000 UTC m=+132.376643374" watchObservedRunningTime="2026-07-21 19:39:29.19223074 +0000 UTC m=+132.376672420" Jul 21 19:39:32.492112 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:32.492080 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:36.350483 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:36.350439 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:36.351000 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:36.350543 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:36.355196 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:36.355174 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:37.184430 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:37.184383 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:37.184620 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:37.184456 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:37.212275 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:37.212242 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:39:37.276444 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:37.276412 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c9ddb8c4-m52qg"] Jul 21 19:39:51.021285 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.021246 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-96ccc48f6-nwb2r" podUID="23d4a22e-1d5e-46ac-be0a-a473d93c102e" containerName="console" containerID="cri-o://11a602f294b181ade2cfc63a46507446cdd4d2ff0f163129603976463c755544" gracePeriod=15 Jul 21 19:39:51.262090 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.262061 2557 generic.go:356] "Generic (PLEG): container finished" podID="d4272620-b668-47a1-b43d-ba5fd58231be" containerID="f84bfd1c2bd0f2b2caa99cec9f56152d4c0d8f75c525d8c7020813ae66d240da" exitCode=0 Jul 21 19:39:51.262205 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.262129 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" event={"ID":"d4272620-b668-47a1-b43d-ba5fd58231be","Type":"ContainerDied","Data":"f84bfd1c2bd0f2b2caa99cec9f56152d4c0d8f75c525d8c7020813ae66d240da"} Jul 21 19:39:51.262528 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.262509 2557 scope.go:117] "RemoveContainer" containerID="f84bfd1c2bd0f2b2caa99cec9f56152d4c0d8f75c525d8c7020813ae66d240da" Jul 21 19:39:51.263506 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.263482 2557 generic.go:356] "Generic (PLEG): container finished" podID="23d4a22e-1d5e-46ac-be0a-a473d93c102e" containerID="11a602f294b181ade2cfc63a46507446cdd4d2ff0f163129603976463c755544" exitCode=0 Jul 21 19:39:51.263610 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.263526 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-96ccc48f6-nwb2r" event={"ID":"23d4a22e-1d5e-46ac-be0a-a473d93c102e","Type":"ContainerDied","Data":"11a602f294b181ade2cfc63a46507446cdd4d2ff0f163129603976463c755544"} Jul 21 19:39:51.304068 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.304046 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:51.446146 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446111 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-config\") pod \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " Jul 21 19:39:51.446146 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446148 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-oauth-serving-cert\") pod \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " Jul 21 19:39:51.446395 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446169 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-serving-cert\") pod \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " Jul 21 19:39:51.446395 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446194 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-service-ca\") pod \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " Jul 21 19:39:51.446395 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446213 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5stj\" (UniqueName: \"kubernetes.io/projected/23d4a22e-1d5e-46ac-be0a-a473d93c102e-kube-api-access-d5stj\") pod \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " Jul 21 19:39:51.446395 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446255 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-oauth-config\") pod \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " Jul 21 19:39:51.446395 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446306 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-trusted-ca-bundle\") pod \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\" (UID: \"23d4a22e-1d5e-46ac-be0a-a473d93c102e\") " Jul 21 19:39:51.446638 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446536 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-config" (OuterVolumeSpecName: "console-config") pod "23d4a22e-1d5e-46ac-be0a-a473d93c102e" (UID: "23d4a22e-1d5e-46ac-be0a-a473d93c102e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:39:51.446638 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446603 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "23d4a22e-1d5e-46ac-be0a-a473d93c102e" (UID: "23d4a22e-1d5e-46ac-be0a-a473d93c102e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:39:51.446638 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446616 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-service-ca" (OuterVolumeSpecName: "service-ca") pod "23d4a22e-1d5e-46ac-be0a-a473d93c102e" (UID: "23d4a22e-1d5e-46ac-be0a-a473d93c102e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:39:51.446842 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.446823 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "23d4a22e-1d5e-46ac-be0a-a473d93c102e" (UID: "23d4a22e-1d5e-46ac-be0a-a473d93c102e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:39:51.448654 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.448628 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d4a22e-1d5e-46ac-be0a-a473d93c102e-kube-api-access-d5stj" (OuterVolumeSpecName: "kube-api-access-d5stj") pod "23d4a22e-1d5e-46ac-be0a-a473d93c102e" (UID: "23d4a22e-1d5e-46ac-be0a-a473d93c102e"). InnerVolumeSpecName "kube-api-access-d5stj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:39:51.448942 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.448923 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "23d4a22e-1d5e-46ac-be0a-a473d93c102e" (UID: "23d4a22e-1d5e-46ac-be0a-a473d93c102e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:39:51.449011 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.448948 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "23d4a22e-1d5e-46ac-be0a-a473d93c102e" (UID: "23d4a22e-1d5e-46ac-be0a-a473d93c102e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:39:51.547215 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.547190 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-config\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:39:51.547215 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.547216 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-oauth-serving-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:39:51.547360 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.547226 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-serving-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:39:51.547360 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.547235 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-service-ca\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:39:51.547360 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.547244 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5stj\" (UniqueName: \"kubernetes.io/projected/23d4a22e-1d5e-46ac-be0a-a473d93c102e-kube-api-access-d5stj\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:39:51.547360 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.547253 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23d4a22e-1d5e-46ac-be0a-a473d93c102e-console-oauth-config\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:39:51.547360 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:51.547261 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23d4a22e-1d5e-46ac-be0a-a473d93c102e-trusted-ca-bundle\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:39:52.268895 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.268795 2557 generic.go:356] "Generic (PLEG): container finished" podID="ac97b694-bcc5-47c3-a779-9bf9b79bbdf0" containerID="f840cdd9cf010be1bb4df95bbfc08885088ff429fad30eb8bd47adb7559e2961" exitCode=0 Jul 21 19:39:52.268895 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.268886 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" event={"ID":"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0","Type":"ContainerDied","Data":"f840cdd9cf010be1bb4df95bbfc08885088ff429fad30eb8bd47adb7559e2961"} Jul 21 19:39:52.269359 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.269212 2557 scope.go:117] "RemoveContainer" containerID="f840cdd9cf010be1bb4df95bbfc08885088ff429fad30eb8bd47adb7559e2961" Jul 21 19:39:52.270357 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.270340 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-96ccc48f6-nwb2r" Jul 21 19:39:52.270467 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.270383 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-96ccc48f6-nwb2r" event={"ID":"23d4a22e-1d5e-46ac-be0a-a473d93c102e","Type":"ContainerDied","Data":"b52b230d9ade28aa9d61c355db1abbb828e40c7b4d1f7c58e7a6fc3cf11ed362"} Jul 21 19:39:52.270467 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.270449 2557 scope.go:117] "RemoveContainer" containerID="11a602f294b181ade2cfc63a46507446cdd4d2ff0f163129603976463c755544" Jul 21 19:39:52.272593 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.272567 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-96z7v" event={"ID":"d4272620-b668-47a1-b43d-ba5fd58231be","Type":"ContainerStarted","Data":"af8ec213a236651ef56a56e460d0aa499d326aa3683388d689c3be067b7fe475"} Jul 21 19:39:52.314782 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.314752 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-96ccc48f6-nwb2r"] Jul 21 19:39:52.323140 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.323111 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-96ccc48f6-nwb2r"] Jul 21 19:39:52.507052 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:52.505434 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23d4a22e-1d5e-46ac-be0a-a473d93c102e" path="/var/lib/kubelet/pods/23d4a22e-1d5e-46ac-be0a-a473d93c102e/volumes" Jul 21 19:39:53.278602 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:53.278563 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7k4hz" event={"ID":"ac97b694-bcc5-47c3-a779-9bf9b79bbdf0","Type":"ContainerStarted","Data":"065c3017b08f50f0747a0d7c00991e9c0e74acab3c91bdb5fe79f30881d40b0b"} Jul 21 19:39:56.292221 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:56.292184 2557 generic.go:356] "Generic (PLEG): container finished" podID="58826593-b4e6-4152-a0a9-a5b9746af2a9" containerID="8c720a819be0318cdef9ed23bdaa202129b31acf31f2cda269c8aef177e95f59" exitCode=0 Jul 21 19:39:56.292579 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:56.292257 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kcl4p" event={"ID":"58826593-b4e6-4152-a0a9-a5b9746af2a9","Type":"ContainerDied","Data":"8c720a819be0318cdef9ed23bdaa202129b31acf31f2cda269c8aef177e95f59"} Jul 21 19:39:56.292637 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:56.292594 2557 scope.go:117] "RemoveContainer" containerID="8c720a819be0318cdef9ed23bdaa202129b31acf31f2cda269c8aef177e95f59" Jul 21 19:39:57.189731 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:57.189702 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:57.193679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:57.193653 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-58ffcbfbc7-6hw54" Jul 21 19:39:57.298717 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:39:57.298677 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kcl4p" event={"ID":"58826593-b4e6-4152-a0a9-a5b9746af2a9","Type":"ContainerStarted","Data":"91a0b8b13ca5edfe3daa269d37b18231654248cc98e9669d68e6115dd3c333cd"} Jul 21 19:40:02.297700 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.297655 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75c9ddb8c4-m52qg" podUID="5c919420-2785-4a17-a4ce-8728aa2ad953" containerName="console" containerID="cri-o://f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6" gracePeriod=15 Jul 21 19:40:02.549079 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.549014 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:40:02.646099 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.646059 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-oauth-serving-cert\") pod \"5c919420-2785-4a17-a4ce-8728aa2ad953\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " Jul 21 19:40:02.646279 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.646139 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-trusted-ca-bundle\") pod \"5c919420-2785-4a17-a4ce-8728aa2ad953\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " Jul 21 19:40:02.646279 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.646166 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-serving-cert\") pod \"5c919420-2785-4a17-a4ce-8728aa2ad953\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " Jul 21 19:40:02.646360 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.646282 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxzjv\" (UniqueName: \"kubernetes.io/projected/5c919420-2785-4a17-a4ce-8728aa2ad953-kube-api-access-lxzjv\") pod \"5c919420-2785-4a17-a4ce-8728aa2ad953\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " Jul 21 19:40:02.646360 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.646316 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-oauth-config\") pod \"5c919420-2785-4a17-a4ce-8728aa2ad953\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " Jul 21 19:40:02.646438 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.646364 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-service-ca\") pod \"5c919420-2785-4a17-a4ce-8728aa2ad953\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " Jul 21 19:40:02.646438 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.646387 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-console-config\") pod \"5c919420-2785-4a17-a4ce-8728aa2ad953\" (UID: \"5c919420-2785-4a17-a4ce-8728aa2ad953\") " Jul 21 19:40:02.646544 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.646523 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5c919420-2785-4a17-a4ce-8728aa2ad953" (UID: "5c919420-2785-4a17-a4ce-8728aa2ad953"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:40:02.646607 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.646567 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5c919420-2785-4a17-a4ce-8728aa2ad953" (UID: "5c919420-2785-4a17-a4ce-8728aa2ad953"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:40:02.647268 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.647213 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-service-ca" (OuterVolumeSpecName: "service-ca") pod "5c919420-2785-4a17-a4ce-8728aa2ad953" (UID: "5c919420-2785-4a17-a4ce-8728aa2ad953"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:40:02.647413 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.647316 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-console-config" (OuterVolumeSpecName: "console-config") pod "5c919420-2785-4a17-a4ce-8728aa2ad953" (UID: "5c919420-2785-4a17-a4ce-8728aa2ad953"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:40:02.647413 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.647385 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-trusted-ca-bundle\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:40:02.647531 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.647414 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-service-ca\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:40:02.647531 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.647429 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-oauth-serving-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:40:02.649985 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.648720 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5c919420-2785-4a17-a4ce-8728aa2ad953" (UID: "5c919420-2785-4a17-a4ce-8728aa2ad953"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:40:02.650537 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.650510 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5c919420-2785-4a17-a4ce-8728aa2ad953" (UID: "5c919420-2785-4a17-a4ce-8728aa2ad953"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:40:02.653260 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.653040 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c919420-2785-4a17-a4ce-8728aa2ad953-kube-api-access-lxzjv" (OuterVolumeSpecName: "kube-api-access-lxzjv") pod "5c919420-2785-4a17-a4ce-8728aa2ad953" (UID: "5c919420-2785-4a17-a4ce-8728aa2ad953"). InnerVolumeSpecName "kube-api-access-lxzjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:40:02.748383 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.748346 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-serving-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:40:02.748540 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.748397 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxzjv\" (UniqueName: \"kubernetes.io/projected/5c919420-2785-4a17-a4ce-8728aa2ad953-kube-api-access-lxzjv\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:40:02.748540 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.748408 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c919420-2785-4a17-a4ce-8728aa2ad953-console-oauth-config\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:40:02.748540 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:02.748418 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c919420-2785-4a17-a4ce-8728aa2ad953-console-config\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:40:03.319858 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:03.319821 2557 generic.go:356] "Generic (PLEG): container finished" podID="5c919420-2785-4a17-a4ce-8728aa2ad953" containerID="f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6" exitCode=0 Jul 21 19:40:03.320254 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:03.319899 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c9ddb8c4-m52qg" Jul 21 19:40:03.320254 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:03.319911 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c9ddb8c4-m52qg" event={"ID":"5c919420-2785-4a17-a4ce-8728aa2ad953","Type":"ContainerDied","Data":"f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6"} Jul 21 19:40:03.320254 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:03.319994 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c9ddb8c4-m52qg" event={"ID":"5c919420-2785-4a17-a4ce-8728aa2ad953","Type":"ContainerDied","Data":"593eddedf5b73b355d0b7e5bddd70c37135b34fd105b0ab68c40f8ceb40a8c4b"} Jul 21 19:40:03.320254 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:03.320022 2557 scope.go:117] "RemoveContainer" containerID="f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6" Jul 21 19:40:03.329118 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:03.329088 2557 scope.go:117] "RemoveContainer" containerID="f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6" Jul 21 19:40:03.329410 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:40:03.329389 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6\": container with ID starting with f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6 not found: ID does not exist" containerID="f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6" Jul 21 19:40:03.329480 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:03.329423 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6"} err="failed to get container status \"f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6\": rpc error: code = NotFound desc = could not find container \"f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6\": container with ID starting with f095686e6a9307baf657a25f0742dcbceaf8093e0d8ca7b8351ffb2640b3d7b6 not found: ID does not exist" Jul 21 19:40:03.340034 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:03.340011 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c9ddb8c4-m52qg"] Jul 21 19:40:03.344418 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:03.344395 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75c9ddb8c4-m52qg"] Jul 21 19:40:04.504251 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:04.504219 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c919420-2785-4a17-a4ce-8728aa2ad953" path="/var/lib/kubelet/pods/5c919420-2785-4a17-a4ce-8728aa2ad953/volumes" Jul 21 19:40:42.352431 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.352395 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-788fc66c5d-dmt65"] Jul 21 19:40:42.353033 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.352695 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d4a22e-1d5e-46ac-be0a-a473d93c102e" containerName="console" Jul 21 19:40:42.353033 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.352708 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d4a22e-1d5e-46ac-be0a-a473d93c102e" containerName="console" Jul 21 19:40:42.353033 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.352735 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c919420-2785-4a17-a4ce-8728aa2ad953" containerName="console" Jul 21 19:40:42.353033 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.352742 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c919420-2785-4a17-a4ce-8728aa2ad953" containerName="console" Jul 21 19:40:42.353033 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.352826 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d4a22e-1d5e-46ac-be0a-a473d93c102e" containerName="console" Jul 21 19:40:42.353033 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.352841 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c919420-2785-4a17-a4ce-8728aa2ad953" containerName="console" Jul 21 19:40:42.355934 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.355913 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.366217 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.366189 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788fc66c5d-dmt65"] Jul 21 19:40:42.476344 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.476313 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-service-ca\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.476344 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.476347 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-config\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.476587 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.476368 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdqg\" (UniqueName: \"kubernetes.io/projected/3899a765-db3b-4ace-a0b2-b4024f1ea317-kube-api-access-qfdqg\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.476587 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.476446 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-oauth-config\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.476587 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.476470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-trusted-ca-bundle\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.476587 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.476493 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-serving-cert\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.476587 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.476541 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-oauth-serving-cert\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.577691 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.577652 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-serving-cert\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.577901 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.577719 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-oauth-serving-cert\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.577901 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.577839 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-service-ca\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.578013 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.577909 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-config\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.578013 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.577941 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdqg\" (UniqueName: \"kubernetes.io/projected/3899a765-db3b-4ace-a0b2-b4024f1ea317-kube-api-access-qfdqg\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.578111 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.578036 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-oauth-config\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.578111 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.578066 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-trusted-ca-bundle\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.578526 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.578494 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-service-ca\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.578628 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.578494 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-oauth-serving-cert\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.578628 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.578567 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-config\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.578867 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.578850 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-trusted-ca-bundle\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.580201 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.580177 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-serving-cert\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.580614 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.580591 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-oauth-config\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.585772 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.585755 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdqg\" (UniqueName: \"kubernetes.io/projected/3899a765-db3b-4ace-a0b2-b4024f1ea317-kube-api-access-qfdqg\") pod \"console-788fc66c5d-dmt65\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.666860 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.666823 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:42.794560 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:42.794529 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788fc66c5d-dmt65"] Jul 21 19:40:43.474072 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:43.474034 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788fc66c5d-dmt65" event={"ID":"3899a765-db3b-4ace-a0b2-b4024f1ea317","Type":"ContainerStarted","Data":"f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac"} Jul 21 19:40:43.474455 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:43.474093 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788fc66c5d-dmt65" event={"ID":"3899a765-db3b-4ace-a0b2-b4024f1ea317","Type":"ContainerStarted","Data":"43c232bae16b3e37a13f19a4d3355c289da719974a483273be31b2fb682cf753"} Jul 21 19:40:43.500785 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:43.500730 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-788fc66c5d-dmt65" podStartSLOduration=1.500706 podStartE2EDuration="1.500706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:40:43.499054046 +0000 UTC m=+206.683495746" watchObservedRunningTime="2026-07-21 19:40:43.500706 +0000 UTC m=+206.685147681" Jul 21 19:40:52.667431 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:52.667386 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:52.667931 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:52.667471 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:52.672227 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:52.672204 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:53.517830 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:53.517800 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:40:53.566450 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:40:53.566416 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-785f8fdb44-t94ff"] Jul 21 19:41:18.588227 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.588185 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-785f8fdb44-t94ff" podUID="793d3b3b-c45e-4397-bf0d-cddb1f665f7d" containerName="console" containerID="cri-o://607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a" gracePeriod=15 Jul 21 19:41:18.827153 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.827129 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:41:18.861531 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.861466 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zwct\" (UniqueName: \"kubernetes.io/projected/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-kube-api-access-2zwct\") pod \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " Jul 21 19:41:18.861659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.861534 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-service-ca\") pod \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " Jul 21 19:41:18.861659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.861564 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-serving-cert\") pod \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " Jul 21 19:41:18.861659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.861602 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-oauth-serving-cert\") pod \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " Jul 21 19:41:18.861659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.861626 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-trusted-ca-bundle\") pod \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " Jul 21 19:41:18.861659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.861654 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-config\") pod \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " Jul 21 19:41:18.861937 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.861688 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-oauth-config\") pod \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\" (UID: \"793d3b3b-c45e-4397-bf0d-cddb1f665f7d\") " Jul 21 19:41:18.862131 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.862079 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "793d3b3b-c45e-4397-bf0d-cddb1f665f7d" (UID: "793d3b3b-c45e-4397-bf0d-cddb1f665f7d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:41:18.862131 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.862088 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-service-ca" (OuterVolumeSpecName: "service-ca") pod "793d3b3b-c45e-4397-bf0d-cddb1f665f7d" (UID: "793d3b3b-c45e-4397-bf0d-cddb1f665f7d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:41:18.862330 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.862227 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-config" (OuterVolumeSpecName: "console-config") pod "793d3b3b-c45e-4397-bf0d-cddb1f665f7d" (UID: "793d3b3b-c45e-4397-bf0d-cddb1f665f7d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:41:18.862330 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.862242 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "793d3b3b-c45e-4397-bf0d-cddb1f665f7d" (UID: "793d3b3b-c45e-4397-bf0d-cddb1f665f7d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:41:18.863770 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.863742 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "793d3b3b-c45e-4397-bf0d-cddb1f665f7d" (UID: "793d3b3b-c45e-4397-bf0d-cddb1f665f7d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:41:18.863856 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.863800 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "793d3b3b-c45e-4397-bf0d-cddb1f665f7d" (UID: "793d3b3b-c45e-4397-bf0d-cddb1f665f7d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:41:18.863856 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.863839 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-kube-api-access-2zwct" (OuterVolumeSpecName: "kube-api-access-2zwct") pod "793d3b3b-c45e-4397-bf0d-cddb1f665f7d" (UID: "793d3b3b-c45e-4397-bf0d-cddb1f665f7d"). InnerVolumeSpecName "kube-api-access-2zwct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:41:18.962750 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.962718 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-oauth-serving-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:41:18.962750 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.962742 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-trusted-ca-bundle\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:41:18.962750 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.962752 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-config\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:41:18.962979 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.962761 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-oauth-config\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:41:18.962979 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.962772 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zwct\" (UniqueName: \"kubernetes.io/projected/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-kube-api-access-2zwct\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:41:18.962979 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.962781 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-service-ca\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:41:18.962979 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:18.962790 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793d3b3b-c45e-4397-bf0d-cddb1f665f7d-console-serving-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:41:19.609748 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:19.609709 2557 generic.go:356] "Generic (PLEG): container finished" podID="793d3b3b-c45e-4397-bf0d-cddb1f665f7d" containerID="607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a" exitCode=0 Jul 21 19:41:19.610214 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:19.609774 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785f8fdb44-t94ff" Jul 21 19:41:19.610214 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:19.609790 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785f8fdb44-t94ff" event={"ID":"793d3b3b-c45e-4397-bf0d-cddb1f665f7d","Type":"ContainerDied","Data":"607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a"} Jul 21 19:41:19.610214 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:19.609841 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785f8fdb44-t94ff" event={"ID":"793d3b3b-c45e-4397-bf0d-cddb1f665f7d","Type":"ContainerDied","Data":"810efe870cd463fe4a915077745cd3269381019ef1849cad3697ead4b286d93c"} Jul 21 19:41:19.610214 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:19.609863 2557 scope.go:117] "RemoveContainer" containerID="607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a" Jul 21 19:41:19.619051 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:19.619034 2557 scope.go:117] "RemoveContainer" containerID="607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a" Jul 21 19:41:19.619305 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:41:19.619288 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a\": container with ID starting with 607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a not found: ID does not exist" containerID="607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a" Jul 21 19:41:19.619380 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:19.619311 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a"} err="failed to get container status \"607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a\": rpc error: code = NotFound desc = could not find container \"607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a\": container with ID starting with 607b3ac94ece562aadc2cfd0b08e975e9f00000b1a2abb6f675a7d8166c0b33a not found: ID does not exist" Jul 21 19:41:19.634051 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:19.634017 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-785f8fdb44-t94ff"] Jul 21 19:41:19.637123 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:19.637098 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-785f8fdb44-t94ff"] Jul 21 19:41:20.503969 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:41:20.503936 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="793d3b3b-c45e-4397-bf0d-cddb1f665f7d" path="/var/lib/kubelet/pods/793d3b3b-c45e-4397-bf0d-cddb1f665f7d/volumes" Jul 21 19:42:16.645010 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.644974 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq"] Jul 21 19:42:16.645526 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.645396 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793d3b3b-c45e-4397-bf0d-cddb1f665f7d" containerName="console" Jul 21 19:42:16.645526 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.645412 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="793d3b3b-c45e-4397-bf0d-cddb1f665f7d" containerName="console" Jul 21 19:42:16.645526 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.645486 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="793d3b3b-c45e-4397-bf0d-cddb1f665f7d" containerName="console" Jul 21 19:42:16.648421 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.648398 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.651362 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.651341 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 19:42:16.652324 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.652307 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 19:42:16.652378 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.652322 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4kcvb\"" Jul 21 19:42:16.659176 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.659155 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq"] Jul 21 19:42:16.749697 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.749654 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.749919 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.749726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9pr7\" (UniqueName: \"kubernetes.io/projected/35fd6b38-423b-4559-b161-4d27c0cd4d34-kube-api-access-l9pr7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.749919 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.749804 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.850963 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.850928 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.850963 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.850977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.851202 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.851028 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9pr7\" (UniqueName: \"kubernetes.io/projected/35fd6b38-423b-4559-b161-4d27c0cd4d34-kube-api-access-l9pr7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.851338 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.851317 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.851424 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.851404 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.860021 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.859989 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9pr7\" (UniqueName: \"kubernetes.io/projected/35fd6b38-423b-4559-b161-4d27c0cd4d34-kube-api-access-l9pr7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.958160 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.958075 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:16.994264 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.994226 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dkh4c"] Jul 21 19:42:16.997530 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:16.997504 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:17.001042 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.001011 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 19:42:17.001173 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.001026 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-95659\"" Jul 21 19:42:17.001173 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.001055 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 19:42:17.006160 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.006136 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dkh4c"] Jul 21 19:42:17.052487 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.052455 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dkh4c\" (UID: \"c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:17.052660 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.052537 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmqs8\" (UniqueName: \"kubernetes.io/projected/c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8-kube-api-access-lmqs8\") pod \"cert-manager-webhook-6486ffd997-dkh4c\" (UID: \"c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:17.093118 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.093097 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq"] Jul 21 19:42:17.095502 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:42:17.095468 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fd6b38_423b_4559_b161_4d27c0cd4d34.slice/crio-a736b0376f7670f5436c06befec497886c6b720355413d147a24258534abed7f WatchSource:0}: Error finding container a736b0376f7670f5436c06befec497886c6b720355413d147a24258534abed7f: Status 404 returned error can't find the container with id a736b0376f7670f5436c06befec497886c6b720355413d147a24258534abed7f Jul 21 19:42:17.112849 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.112824 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:42:17.117765 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.117731 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:42:17.129026 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.128998 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 19:42:17.153233 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.153197 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dkh4c\" (UID: \"c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:17.153379 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.153249 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmqs8\" (UniqueName: \"kubernetes.io/projected/c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8-kube-api-access-lmqs8\") pod \"cert-manager-webhook-6486ffd997-dkh4c\" (UID: \"c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:17.166953 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.166918 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dkh4c\" (UID: \"c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:17.167225 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.167208 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 19:42:17.178182 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.178154 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 19:42:17.188747 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.188679 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmqs8\" (UniqueName: \"kubernetes.io/projected/c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8-kube-api-access-lmqs8\") pod \"cert-manager-webhook-6486ffd997-dkh4c\" (UID: \"c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:17.318938 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.318772 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-95659\"" Jul 21 19:42:17.331247 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.326590 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:17.481553 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.481527 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dkh4c"] Jul 21 19:42:17.484000 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:42:17.483976 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2fb783c_2ea2_4b77_b6ee_0745fe05c7c8.slice/crio-4bfa4465a88c173ae1a23b6ea462a93e9399045d2e627b39ec0543f3aaf31fef WatchSource:0}: Error finding container 4bfa4465a88c173ae1a23b6ea462a93e9399045d2e627b39ec0543f3aaf31fef: Status 404 returned error can't find the container with id 4bfa4465a88c173ae1a23b6ea462a93e9399045d2e627b39ec0543f3aaf31fef Jul 21 19:42:17.828223 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.828182 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" event={"ID":"35fd6b38-423b-4559-b161-4d27c0cd4d34","Type":"ContainerStarted","Data":"a736b0376f7670f5436c06befec497886c6b720355413d147a24258534abed7f"} Jul 21 19:42:17.829591 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:17.829567 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" event={"ID":"c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8","Type":"ContainerStarted","Data":"4bfa4465a88c173ae1a23b6ea462a93e9399045d2e627b39ec0543f3aaf31fef"} Jul 21 19:42:18.900802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:18.900761 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7"] Jul 21 19:42:18.904255 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:18.904235 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" Jul 21 19:42:18.926923 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:18.926890 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7"] Jul 21 19:42:18.968239 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:18.968206 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg5sb\" (UniqueName: \"kubernetes.io/projected/df5bc41e-f2a8-4d11-985d-be8ad2d8a94e-kube-api-access-qg5sb\") pod \"cert-manager-cainjector-559cdd8c87-6fmw7\" (UID: \"df5bc41e-f2a8-4d11-985d-be8ad2d8a94e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" Jul 21 19:42:18.968411 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:18.968259 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df5bc41e-f2a8-4d11-985d-be8ad2d8a94e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6fmw7\" (UID: \"df5bc41e-f2a8-4d11-985d-be8ad2d8a94e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" Jul 21 19:42:19.069356 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:19.069260 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5sb\" (UniqueName: \"kubernetes.io/projected/df5bc41e-f2a8-4d11-985d-be8ad2d8a94e-kube-api-access-qg5sb\") pod \"cert-manager-cainjector-559cdd8c87-6fmw7\" (UID: \"df5bc41e-f2a8-4d11-985d-be8ad2d8a94e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" Jul 21 19:42:19.069356 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:19.069320 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df5bc41e-f2a8-4d11-985d-be8ad2d8a94e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6fmw7\" (UID: \"df5bc41e-f2a8-4d11-985d-be8ad2d8a94e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" Jul 21 19:42:19.083226 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:19.083196 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df5bc41e-f2a8-4d11-985d-be8ad2d8a94e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6fmw7\" (UID: \"df5bc41e-f2a8-4d11-985d-be8ad2d8a94e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" Jul 21 19:42:19.088729 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:19.088693 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg5sb\" (UniqueName: \"kubernetes.io/projected/df5bc41e-f2a8-4d11-985d-be8ad2d8a94e-kube-api-access-qg5sb\") pod \"cert-manager-cainjector-559cdd8c87-6fmw7\" (UID: \"df5bc41e-f2a8-4d11-985d-be8ad2d8a94e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" Jul 21 19:42:19.220912 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:19.220816 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" Jul 21 19:42:23.109811 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.109785 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7"] Jul 21 19:42:23.130739 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:42:23.130705 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf5bc41e_f2a8_4d11_985d_be8ad2d8a94e.slice/crio-e4a197791039518a998a30bb2bb64b15b30779cb6a62bcfb65fd352235a6e720 WatchSource:0}: Error finding container e4a197791039518a998a30bb2bb64b15b30779cb6a62bcfb65fd352235a6e720: Status 404 returned error can't find the container with id e4a197791039518a998a30bb2bb64b15b30779cb6a62bcfb65fd352235a6e720 Jul 21 19:42:23.859665 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.859622 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" event={"ID":"c2fb783c-2ea2-4b77-b6ee-0745fe05c7c8","Type":"ContainerStarted","Data":"067a7b247db5510887326edff1bb1e10ad0d486888367950d7da5cad348997d4"} Jul 21 19:42:23.859870 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.859764 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:23.861372 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.861347 2557 generic.go:356] "Generic (PLEG): container finished" podID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerID="f31f492780cc8be1db4ab4f3fb27911ce5cbd8312b4bc8deedd86611c595ea50" exitCode=0 Jul 21 19:42:23.861488 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.861378 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" event={"ID":"35fd6b38-423b-4559-b161-4d27c0cd4d34","Type":"ContainerDied","Data":"f31f492780cc8be1db4ab4f3fb27911ce5cbd8312b4bc8deedd86611c595ea50"} Jul 21 19:42:23.862537 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.862518 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 19:42:23.862969 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.862941 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" event={"ID":"df5bc41e-f2a8-4d11-985d-be8ad2d8a94e","Type":"ContainerStarted","Data":"f23c232bc637c59bd2a571020d93846863b8205d41daf472f834fb7ce21ef386"} Jul 21 19:42:23.863058 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.862978 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" event={"ID":"df5bc41e-f2a8-4d11-985d-be8ad2d8a94e","Type":"ContainerStarted","Data":"e4a197791039518a998a30bb2bb64b15b30779cb6a62bcfb65fd352235a6e720"} Jul 21 19:42:23.881788 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.881746 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" podStartSLOduration=2.397510141 podStartE2EDuration="7.881732384s" totalImagesPullingTime="5.484222243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:42:23.880576242 +0000 UTC m=+307.065017920" watchObservedRunningTime="2026-07-21 19:42:23.881732384 +0000 UTC m=+307.066174064" Jul 21 19:42:23.904704 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:23.904657 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6fmw7" podStartSLOduration=5.904639235 podStartE2EDuration="5.904639235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:42:23.903693728 +0000 UTC m=+307.088135406" watchObservedRunningTime="2026-07-21 19:42:23.904639235 +0000 UTC m=+307.089080916" Jul 21 19:42:25.872513 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:25.872430 2557 generic.go:356] "Generic (PLEG): container finished" podID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerID="b81135bf81ed747941fe055107ef5083fe064cbd72e8fb6e02f96985dcd704c3" exitCode=0 Jul 21 19:42:25.872513 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:25.872472 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" event={"ID":"35fd6b38-423b-4559-b161-4d27c0cd4d34","Type":"ContainerDied","Data":"b81135bf81ed747941fe055107ef5083fe064cbd72e8fb6e02f96985dcd704c3"} Jul 21 19:42:29.870179 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:29.870141 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dkh4c" Jul 21 19:42:31.902228 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:31.902191 2557 generic.go:356] "Generic (PLEG): container finished" podID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerID="85c0205966f0372002e025c8403ea6f35665fca17217d8a7295c62e94835ca47" exitCode=0 Jul 21 19:42:31.902667 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:31.902265 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" event={"ID":"35fd6b38-423b-4559-b161-4d27c0cd4d34","Type":"ContainerDied","Data":"85c0205966f0372002e025c8403ea6f35665fca17217d8a7295c62e94835ca47"} Jul 21 19:42:33.032811 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.032781 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:33.098598 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.098562 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-util\") pod \"35fd6b38-423b-4559-b161-4d27c0cd4d34\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " Jul 21 19:42:33.098775 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.098626 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9pr7\" (UniqueName: \"kubernetes.io/projected/35fd6b38-423b-4559-b161-4d27c0cd4d34-kube-api-access-l9pr7\") pod \"35fd6b38-423b-4559-b161-4d27c0cd4d34\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " Jul 21 19:42:33.098775 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.098682 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-bundle\") pod \"35fd6b38-423b-4559-b161-4d27c0cd4d34\" (UID: \"35fd6b38-423b-4559-b161-4d27c0cd4d34\") " Jul 21 19:42:33.099125 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.099101 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-bundle" (OuterVolumeSpecName: "bundle") pod "35fd6b38-423b-4559-b161-4d27c0cd4d34" (UID: "35fd6b38-423b-4559-b161-4d27c0cd4d34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:42:33.100854 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.100826 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fd6b38-423b-4559-b161-4d27c0cd4d34-kube-api-access-l9pr7" (OuterVolumeSpecName: "kube-api-access-l9pr7") pod "35fd6b38-423b-4559-b161-4d27c0cd4d34" (UID: "35fd6b38-423b-4559-b161-4d27c0cd4d34"). InnerVolumeSpecName "kube-api-access-l9pr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:42:33.102755 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.102723 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-util" (OuterVolumeSpecName: "util") pod "35fd6b38-423b-4559-b161-4d27c0cd4d34" (UID: "35fd6b38-423b-4559-b161-4d27c0cd4d34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:42:33.199371 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.199279 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-bundle\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:42:33.199371 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.199319 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35fd6b38-423b-4559-b161-4d27c0cd4d34-util\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:42:33.199371 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.199329 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9pr7\" (UniqueName: \"kubernetes.io/projected/35fd6b38-423b-4559-b161-4d27c0cd4d34-kube-api-access-l9pr7\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:42:33.911289 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.911255 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" Jul 21 19:42:33.911289 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.911262 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnfngq" event={"ID":"35fd6b38-423b-4559-b161-4d27c0cd4d34","Type":"ContainerDied","Data":"a736b0376f7670f5436c06befec497886c6b720355413d147a24258534abed7f"} Jul 21 19:42:33.911487 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:42:33.911302 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a736b0376f7670f5436c06befec497886c6b720355413d147a24258534abed7f" Jul 21 19:43:03.171504 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.171472 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx"] Jul 21 19:43:03.171990 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.171756 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerName="pull" Jul 21 19:43:03.171990 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.171767 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerName="pull" Jul 21 19:43:03.171990 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.171778 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerName="util" Jul 21 19:43:03.171990 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.171783 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerName="util" Jul 21 19:43:03.171990 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.171795 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerName="extract" Jul 21 19:43:03.171990 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.171801 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerName="extract" Jul 21 19:43:03.171990 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.171853 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="35fd6b38-423b-4559-b161-4d27c0cd4d34" containerName="extract" Jul 21 19:43:03.173802 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.173785 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.180330 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.180224 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 19:43:03.180489 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.180470 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 19:43:03.180705 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.180678 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 19:43:03.180825 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.180758 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-brd8z\"" Jul 21 19:43:03.181002 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.180986 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 19:43:03.202620 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.202588 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx"] Jul 21 19:43:03.242630 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.242599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2d5f7f6-4496-4a99-98da-84291973fa06-manifests\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.242630 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.242631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2d5f7f6-4496-4a99-98da-84291973fa06-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.242824 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.242650 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d5f7f6-4496-4a99-98da-84291973fa06-tmp\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.242824 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.242675 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2sfd\" (UniqueName: \"kubernetes.io/projected/f2d5f7f6-4496-4a99-98da-84291973fa06-kube-api-access-s2sfd\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.242824 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.242724 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2d5f7f6-4496-4a99-98da-84291973fa06-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.343703 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.343665 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2d5f7f6-4496-4a99-98da-84291973fa06-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.343937 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.343749 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2d5f7f6-4496-4a99-98da-84291973fa06-manifests\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.343937 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.343770 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2d5f7f6-4496-4a99-98da-84291973fa06-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.343937 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.343787 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d5f7f6-4496-4a99-98da-84291973fa06-tmp\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.343937 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.343806 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2sfd\" (UniqueName: \"kubernetes.io/projected/f2d5f7f6-4496-4a99-98da-84291973fa06-kube-api-access-s2sfd\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.344299 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.344270 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2d5f7f6-4496-4a99-98da-84291973fa06-manifests\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.346070 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.346038 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d5f7f6-4496-4a99-98da-84291973fa06-tmp\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.346266 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.346248 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2d5f7f6-4496-4a99-98da-84291973fa06-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.346308 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.346273 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2d5f7f6-4496-4a99-98da-84291973fa06-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.355525 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.355501 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2sfd\" (UniqueName: \"kubernetes.io/projected/f2d5f7f6-4496-4a99-98da-84291973fa06-kube-api-access-s2sfd\") pod \"opendatahub-operator-controller-manager-7f74b88589-f9jgx\" (UID: \"f2d5f7f6-4496-4a99-98da-84291973fa06\") " pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.484012 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.483924 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:03.656652 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:03.656482 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx"] Jul 21 19:43:03.659567 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:43:03.659541 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d5f7f6_4496_4a99_98da_84291973fa06.slice/crio-db2286f1b84d8db4fc06f49b5abfea089d4a3aad5db6bf3e3a557ad91b1580ee WatchSource:0}: Error finding container db2286f1b84d8db4fc06f49b5abfea089d4a3aad5db6bf3e3a557ad91b1580ee: Status 404 returned error can't find the container with id db2286f1b84d8db4fc06f49b5abfea089d4a3aad5db6bf3e3a557ad91b1580ee Jul 21 19:43:04.027747 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:04.027702 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" event={"ID":"f2d5f7f6-4496-4a99-98da-84291973fa06","Type":"ContainerStarted","Data":"db2286f1b84d8db4fc06f49b5abfea089d4a3aad5db6bf3e3a557ad91b1580ee"} Jul 21 19:43:07.044945 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:07.044902 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" event={"ID":"f2d5f7f6-4496-4a99-98da-84291973fa06","Type":"ContainerStarted","Data":"b0d01cee8a4e79e6974b9d4b40d9dc72d77ee8ed827574404dc3524bd356f6a4"} Jul 21 19:43:08.051509 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:08.051474 2557 generic.go:356] "Generic (PLEG): container finished" podID="f2d5f7f6-4496-4a99-98da-84291973fa06" containerID="b0d01cee8a4e79e6974b9d4b40d9dc72d77ee8ed827574404dc3524bd356f6a4" exitCode=0 Jul 21 19:43:08.051932 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:08.051559 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" event={"ID":"f2d5f7f6-4496-4a99-98da-84291973fa06","Type":"ContainerDied","Data":"b0d01cee8a4e79e6974b9d4b40d9dc72d77ee8ed827574404dc3524bd356f6a4"} Jul 21 19:43:09.057373 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:09.057342 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" event={"ID":"f2d5f7f6-4496-4a99-98da-84291973fa06","Type":"ContainerStarted","Data":"896bbb79d2a532bb6ad0c89afc62120b36a41cde0ffcf10feaf3805ae8e74975"} Jul 21 19:43:09.057824 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:09.057453 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:09.092163 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:09.092115 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" podStartSLOduration=1.513864195 podStartE2EDuration="6.092096813s" totalImagesPullingTime="3.543655712s" totalInitContainerRuntime="1.034576906s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:43:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:43:09.090943858 +0000 UTC m=+352.275385581" watchObservedRunningTime="2026-07-21 19:43:09.092096813 +0000 UTC m=+352.276538495" Jul 21 19:43:20.063776 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:20.063748 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f74b88589-f9jgx" Jul 21 19:43:23.309847 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.309812 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6"] Jul 21 19:43:23.320782 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.320754 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.335731 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.335675 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 19:43:23.335861 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.335842 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 19:43:23.335928 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.335842 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 19:43:23.335969 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.335934 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 19:43:23.344032 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.344011 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6"] Jul 21 19:43:23.417828 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.417790 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b5193a-3e53-426b-9ac1-7dd11d70d8e9-tls-certs\") pod \"kube-auth-proxy-7db7f7c8d6-8b2m6\" (UID: \"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9\") " pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.418004 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.417844 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqqz7\" (UniqueName: \"kubernetes.io/projected/d4b5193a-3e53-426b-9ac1-7dd11d70d8e9-kube-api-access-bqqz7\") pod \"kube-auth-proxy-7db7f7c8d6-8b2m6\" (UID: \"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9\") " pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.418004 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.417926 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b5193a-3e53-426b-9ac1-7dd11d70d8e9-tmp\") pod \"kube-auth-proxy-7db7f7c8d6-8b2m6\" (UID: \"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9\") " pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.518355 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.518326 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b5193a-3e53-426b-9ac1-7dd11d70d8e9-tls-certs\") pod \"kube-auth-proxy-7db7f7c8d6-8b2m6\" (UID: \"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9\") " pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.518521 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.518373 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqqz7\" (UniqueName: \"kubernetes.io/projected/d4b5193a-3e53-426b-9ac1-7dd11d70d8e9-kube-api-access-bqqz7\") pod \"kube-auth-proxy-7db7f7c8d6-8b2m6\" (UID: \"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9\") " pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.518521 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.518410 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b5193a-3e53-426b-9ac1-7dd11d70d8e9-tmp\") pod \"kube-auth-proxy-7db7f7c8d6-8b2m6\" (UID: \"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9\") " pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.520578 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.520552 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b5193a-3e53-426b-9ac1-7dd11d70d8e9-tmp\") pod \"kube-auth-proxy-7db7f7c8d6-8b2m6\" (UID: \"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9\") " pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.520707 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.520689 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b5193a-3e53-426b-9ac1-7dd11d70d8e9-tls-certs\") pod \"kube-auth-proxy-7db7f7c8d6-8b2m6\" (UID: \"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9\") " pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.528288 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.528265 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-k86fn"] Jul 21 19:43:23.531616 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.531595 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:23.541921 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.541900 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 19:43:23.544115 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.544091 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqqz7\" (UniqueName: \"kubernetes.io/projected/d4b5193a-3e53-426b-9ac1-7dd11d70d8e9-kube-api-access-bqqz7\") pod \"kube-auth-proxy-7db7f7c8d6-8b2m6\" (UID: \"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9\") " pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.551251 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:43:23.551224 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"odh-model-controller-7998c67dc4-k86fn\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c" pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:23.551496 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:43:23.551477 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-dockercfg-6nwdl\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6nwdl\"" type="*v1.Secret" Jul 21 19:43:23.591819 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.591738 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-k86fn"] Jul 21 19:43:23.618937 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.618897 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv2p6\" (UniqueName: \"kubernetes.io/projected/9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c-kube-api-access-vv2p6\") pod \"odh-model-controller-7998c67dc4-k86fn\" (UID: \"9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c\") " pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:23.619149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.618952 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c-cert\") pod \"odh-model-controller-7998c67dc4-k86fn\" (UID: \"9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c\") " pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:23.630460 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.630433 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" Jul 21 19:43:23.720444 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.720416 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2p6\" (UniqueName: \"kubernetes.io/projected/9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c-kube-api-access-vv2p6\") pod \"odh-model-controller-7998c67dc4-k86fn\" (UID: \"9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c\") " pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:23.720590 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.720461 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c-cert\") pod \"odh-model-controller-7998c67dc4-k86fn\" (UID: \"9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c\") " pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:23.720698 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:43:23.720675 2557 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 19:43:23.720791 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:43:23.720763 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c-cert podName:9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c nodeName:}" failed. No retries permitted until 2026-07-21 19:43:24.22073851 +0000 UTC m=+367.405180172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c-cert") pod "odh-model-controller-7998c67dc4-k86fn" (UID: "9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c") : secret "odh-model-controller-webhook-cert" not found Jul 21 19:43:23.741636 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.741600 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2p6\" (UniqueName: \"kubernetes.io/projected/9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c-kube-api-access-vv2p6\") pod \"odh-model-controller-7998c67dc4-k86fn\" (UID: \"9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c\") " pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:23.815303 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:23.815269 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6"] Jul 21 19:43:23.819197 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:43:23.819169 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b5193a_3e53_426b_9ac1_7dd11d70d8e9.slice/crio-1b03943ce9086e1c18192df1b238ca7b84f9acdcc4aa1f6d21bfd2af7b0c8e89 WatchSource:0}: Error finding container 1b03943ce9086e1c18192df1b238ca7b84f9acdcc4aa1f6d21bfd2af7b0c8e89: Status 404 returned error can't find the container with id 1b03943ce9086e1c18192df1b238ca7b84f9acdcc4aa1f6d21bfd2af7b0c8e89 Jul 21 19:43:24.121064 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.121026 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" event={"ID":"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9","Type":"ContainerStarted","Data":"1b03943ce9086e1c18192df1b238ca7b84f9acdcc4aa1f6d21bfd2af7b0c8e89"} Jul 21 19:43:24.225417 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.225375 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c-cert\") pod \"odh-model-controller-7998c67dc4-k86fn\" (UID: \"9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c\") " pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:24.227787 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.227768 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c-cert\") pod \"odh-model-controller-7998c67dc4-k86fn\" (UID: \"9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c\") " pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:24.713608 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.713565 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6nwdl\"" Jul 21 19:43:24.722934 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.722908 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:24.750104 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.750073 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lzs2n"] Jul 21 19:43:24.753833 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.753782 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:43:24.757729 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.757704 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 19:43:24.767135 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.766936 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-kcclr\"" Jul 21 19:43:24.792392 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.784556 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lzs2n"] Jul 21 19:43:24.831066 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.831029 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f7kq\" (UniqueName: \"kubernetes.io/projected/e4a6f6d5-ad74-4912-8a89-d4427ac74e23-kube-api-access-7f7kq\") pod \"model-serving-api-54d89bd5fc-lzs2n\" (UID: \"e4a6f6d5-ad74-4912-8a89-d4427ac74e23\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:43:24.831249 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.831089 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4a6f6d5-ad74-4912-8a89-d4427ac74e23-tls-certs\") pod \"model-serving-api-54d89bd5fc-lzs2n\" (UID: \"e4a6f6d5-ad74-4912-8a89-d4427ac74e23\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:43:24.904371 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.904343 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-k86fn"] Jul 21 19:43:24.907142 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:43:24.907106 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc7e64e_9e32_4a8c_a3a4_a30c6bfb823c.slice/crio-11c4f0821c94f8bf87f0f23c535e30bac038492aaa0cd36091e136f8ecef35b4 WatchSource:0}: Error finding container 11c4f0821c94f8bf87f0f23c535e30bac038492aaa0cd36091e136f8ecef35b4: Status 404 returned error can't find the container with id 11c4f0821c94f8bf87f0f23c535e30bac038492aaa0cd36091e136f8ecef35b4 Jul 21 19:43:24.931755 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.931729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4a6f6d5-ad74-4912-8a89-d4427ac74e23-tls-certs\") pod \"model-serving-api-54d89bd5fc-lzs2n\" (UID: \"e4a6f6d5-ad74-4912-8a89-d4427ac74e23\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:43:24.931936 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.931848 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f7kq\" (UniqueName: \"kubernetes.io/projected/e4a6f6d5-ad74-4912-8a89-d4427ac74e23-kube-api-access-7f7kq\") pod \"model-serving-api-54d89bd5fc-lzs2n\" (UID: \"e4a6f6d5-ad74-4912-8a89-d4427ac74e23\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:43:24.934475 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.934450 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4a6f6d5-ad74-4912-8a89-d4427ac74e23-tls-certs\") pod \"model-serving-api-54d89bd5fc-lzs2n\" (UID: \"e4a6f6d5-ad74-4912-8a89-d4427ac74e23\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:43:24.948693 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:24.948668 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f7kq\" (UniqueName: \"kubernetes.io/projected/e4a6f6d5-ad74-4912-8a89-d4427ac74e23-kube-api-access-7f7kq\") pod \"model-serving-api-54d89bd5fc-lzs2n\" (UID: \"e4a6f6d5-ad74-4912-8a89-d4427ac74e23\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:43:25.067566 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.067491 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:43:25.129522 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.129490 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" event={"ID":"9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c","Type":"ContainerStarted","Data":"11c4f0821c94f8bf87f0f23c535e30bac038492aaa0cd36091e136f8ecef35b4"} Jul 21 19:43:25.210312 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.210280 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lzs2n"] Jul 21 19:43:25.212007 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:43:25.211952 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a6f6d5_ad74_4912_8a89_d4427ac74e23.slice/crio-0824ab4b997eb5111e57e38103090bd090f8848e9eb8d22f74de79f57c13c12b WatchSource:0}: Error finding container 0824ab4b997eb5111e57e38103090bd090f8848e9eb8d22f74de79f57c13c12b: Status 404 returned error can't find the container with id 0824ab4b997eb5111e57e38103090bd090f8848e9eb8d22f74de79f57c13c12b Jul 21 19:43:25.486508 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.486464 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-shf98"] Jul 21 19:43:25.490278 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.490251 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.493584 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.493558 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 19:43:25.493825 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.493804 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6xsbl\"" Jul 21 19:43:25.494366 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.494348 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 19:43:25.494457 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.494372 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 19:43:25.494512 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.494463 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 19:43:25.506869 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.506811 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-shf98"] Jul 21 19:43:25.638748 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.638648 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-token\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.638748 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.638696 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgtvj\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-kube-api-access-hgtvj\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.638748 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.638736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.639181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.638774 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-cacerts\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.639181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.638806 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.639181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.638951 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e19609d-ee10-4caa-b77b-b55621d8f5fd-local-certs\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.639181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.638983 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.740038 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.739942 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-token\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.740038 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.740007 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtvj\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-kube-api-access-hgtvj\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.740556 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.740043 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.740556 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.740079 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-cacerts\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.740556 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.740107 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.740556 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.740180 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e19609d-ee10-4caa-b77b-b55621d8f5fd-local-certs\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.740556 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.740210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.741244 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.741215 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.743925 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.743453 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-cacerts\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.746183 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.745399 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e19609d-ee10-4caa-b77b-b55621d8f5fd-local-certs\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.746183 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.745733 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.746935 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.746912 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.759233 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.759208 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-token\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.759842 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.759820 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtvj\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-kube-api-access-hgtvj\") pod \"istiod-openshift-gateway-7574459656-shf98\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:25.805316 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:25.804999 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:26.138901 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:26.138607 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" event={"ID":"e4a6f6d5-ad74-4912-8a89-d4427ac74e23","Type":"ContainerStarted","Data":"0824ab4b997eb5111e57e38103090bd090f8848e9eb8d22f74de79f57c13c12b"} Jul 21 19:43:28.403442 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:28.403415 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-shf98"] Jul 21 19:43:28.405738 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:43:28.405711 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e19609d_ee10_4caa_b77b_b55621d8f5fd.slice/crio-a6df64fb7318446fa4afd18487d5b221ab6dda6b811866d19e9db8fab1ccc2fe WatchSource:0}: Error finding container a6df64fb7318446fa4afd18487d5b221ab6dda6b811866d19e9db8fab1ccc2fe: Status 404 returned error can't find the container with id a6df64fb7318446fa4afd18487d5b221ab6dda6b811866d19e9db8fab1ccc2fe Jul 21 19:43:29.154552 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:29.154510 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" event={"ID":"7e19609d-ee10-4caa-b77b-b55621d8f5fd","Type":"ContainerStarted","Data":"a6df64fb7318446fa4afd18487d5b221ab6dda6b811866d19e9db8fab1ccc2fe"} Jul 21 19:43:29.156167 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:29.156115 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" event={"ID":"e4a6f6d5-ad74-4912-8a89-d4427ac74e23","Type":"ContainerStarted","Data":"65dc6164614f4e77ba3483a437bcb2995db026b5d6e67f320044968790484453"} Jul 21 19:43:29.156320 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:29.156282 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:43:29.157568 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:29.157527 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" event={"ID":"d4b5193a-3e53-426b-9ac1-7dd11d70d8e9","Type":"ContainerStarted","Data":"ab5a4103fa257febecb2ebedb57e2144cb0b5f03566d7125d09a75c05bf9b0c4"} Jul 21 19:43:29.158907 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:29.158869 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" event={"ID":"9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c","Type":"ContainerStarted","Data":"099f53466057895e90616367db75479e8cc47310334765a68b3236b515eb1f8f"} Jul 21 19:43:29.159021 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:29.159007 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:29.179068 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:29.179021 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" podStartSLOduration=2.1400976959999998 podStartE2EDuration="5.179008047s" totalImagesPullingTime="3.038910351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:43:29.176752013 +0000 UTC m=+372.361193699" watchObservedRunningTime="2026-07-21 19:43:29.179008047 +0000 UTC m=+372.363449720" Jul 21 19:43:29.194265 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:29.194219 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7db7f7c8d6-8b2m6" podStartSLOduration=1.765198802 podStartE2EDuration="6.194206409s" totalImagesPullingTime="4.429007607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:43:29.193515596 +0000 UTC m=+372.377957304" watchObservedRunningTime="2026-07-21 19:43:29.194206409 +0000 UTC m=+372.378648147" Jul 21 19:43:29.215787 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:29.215741 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" podStartSLOduration=2.869148937 podStartE2EDuration="6.215725863s" totalImagesPullingTime="3.346576926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:43:29.214004341 +0000 UTC m=+372.398446044" watchObservedRunningTime="2026-07-21 19:43:29.215725863 +0000 UTC m=+372.400167585" Jul 21 19:43:30.992574 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:30.992539 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:43:30.992832 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:30.992608 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:43:31.170132 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:31.170097 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" event={"ID":"7e19609d-ee10-4caa-b77b-b55621d8f5fd","Type":"ContainerStarted","Data":"9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2"} Jul 21 19:43:31.170312 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:31.170165 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:31.199740 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:31.199684 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" podStartSLOduration=3.615047164 podStartE2EDuration="6.199667276s" totalImagesPullingTime="2.584620112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:43:31.199594618 +0000 UTC m=+374.384036320" watchObservedRunningTime="2026-07-21 19:43:31.199667276 +0000 UTC m=+374.384108957" Jul 21 19:43:32.071016 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.070981 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj"] Jul 21 19:43:32.074365 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.074348 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.079714 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.079691 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-z6hpx\"" Jul 21 19:43:32.092193 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.092167 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj"] Jul 21 19:43:32.176501 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.176472 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:32.199959 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.199930 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.200181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.199964 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.200181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.199985 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.200181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.200061 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.200181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.200076 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.200181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.200102 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.200181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.200147 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlrml\" (UniqueName: \"kubernetes.io/projected/2ffb7861-e586-4d4b-afe4-67165d82f0b0-kube-api-access-jlrml\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.200452 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.200220 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.200452 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.200259 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301241 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301212 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301241 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301243 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301493 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301493 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301333 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301493 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301493 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301454 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301493 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301482 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlrml\" (UniqueName: \"kubernetes.io/projected/2ffb7861-e586-4d4b-afe4-67165d82f0b0-kube-api-access-jlrml\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301781 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301559 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301781 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301636 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.301781 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.301661 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.302083 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.302053 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.302172 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.302124 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.302237 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.302199 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.302237 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.302207 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.303787 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.303755 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.304001 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.303984 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.328799 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.328735 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlrml\" (UniqueName: \"kubernetes.io/projected/2ffb7861-e586-4d4b-afe4-67165d82f0b0-kube-api-access-jlrml\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.329747 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.329724 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffb7861-e586-4d4b-afe4-67165d82f0b0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rrvcj\" (UID: \"2ffb7861-e586-4d4b-afe4-67165d82f0b0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.384510 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.384485 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:32.552172 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:32.552148 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj"] Jul 21 19:43:32.553680 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:43:32.553653 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffb7861_e586_4d4b_afe4_67165d82f0b0.slice/crio-466805f11c2615f89f13a0f0bc2d66bb804b05d5cc2df0b4a90ef633d403253f WatchSource:0}: Error finding container 466805f11c2615f89f13a0f0bc2d66bb804b05d5cc2df0b4a90ef633d403253f: Status 404 returned error can't find the container with id 466805f11c2615f89f13a0f0bc2d66bb804b05d5cc2df0b4a90ef633d403253f Jul 21 19:43:33.179462 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:33.179428 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" event={"ID":"2ffb7861-e586-4d4b-afe4-67165d82f0b0","Type":"ContainerStarted","Data":"466805f11c2615f89f13a0f0bc2d66bb804b05d5cc2df0b4a90ef633d403253f"} Jul 21 19:43:34.606835 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.606791 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz"] Jul 21 19:43:34.613795 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.613771 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-84kp2"] Jul 21 19:43:34.613958 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.613932 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:34.619316 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.619272 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:34.619516 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.619405 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 19:43:34.619581 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.619522 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vdvmc\"" Jul 21 19:43:34.623062 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.623031 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 19:43:34.623190 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.623075 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fncrq\"" Jul 21 19:43:34.637322 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.637297 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-84kp2"] Jul 21 19:43:34.643846 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.643694 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz"] Jul 21 19:43:34.728249 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.728197 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55620973-5262-462b-bbd4-a9f22b9d6f7e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tc5xz\" (UID: \"55620973-5262-462b-bbd4-a9f22b9d6f7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:34.728433 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.728342 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g42xz\" (UniqueName: \"kubernetes.io/projected/55620973-5262-462b-bbd4-a9f22b9d6f7e-kube-api-access-g42xz\") pod \"llmisvc-controller-manager-777c6bd7d5-tc5xz\" (UID: \"55620973-5262-462b-bbd4-a9f22b9d6f7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:34.728516 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.728439 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cnfs\" (UniqueName: \"kubernetes.io/projected/3956c1ef-0b89-4e91-a16d-d08fa3f87476-kube-api-access-9cnfs\") pod \"kserve-controller-manager-d6f4b65b8-84kp2\" (UID: \"3956c1ef-0b89-4e91-a16d-d08fa3f87476\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:34.728516 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.728476 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3956c1ef-0b89-4e91-a16d-d08fa3f87476-cert\") pod \"kserve-controller-manager-d6f4b65b8-84kp2\" (UID: \"3956c1ef-0b89-4e91-a16d-d08fa3f87476\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:34.829184 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.829145 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55620973-5262-462b-bbd4-a9f22b9d6f7e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tc5xz\" (UID: \"55620973-5262-462b-bbd4-a9f22b9d6f7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:34.829373 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.829231 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g42xz\" (UniqueName: \"kubernetes.io/projected/55620973-5262-462b-bbd4-a9f22b9d6f7e-kube-api-access-g42xz\") pod \"llmisvc-controller-manager-777c6bd7d5-tc5xz\" (UID: \"55620973-5262-462b-bbd4-a9f22b9d6f7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:34.829373 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.829284 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cnfs\" (UniqueName: \"kubernetes.io/projected/3956c1ef-0b89-4e91-a16d-d08fa3f87476-kube-api-access-9cnfs\") pod \"kserve-controller-manager-d6f4b65b8-84kp2\" (UID: \"3956c1ef-0b89-4e91-a16d-d08fa3f87476\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:34.829373 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.829308 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3956c1ef-0b89-4e91-a16d-d08fa3f87476-cert\") pod \"kserve-controller-manager-d6f4b65b8-84kp2\" (UID: \"3956c1ef-0b89-4e91-a16d-d08fa3f87476\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:34.829373 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:43:34.829320 2557 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 21 19:43:34.829567 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:43:34.829399 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55620973-5262-462b-bbd4-a9f22b9d6f7e-cert podName:55620973-5262-462b-bbd4-a9f22b9d6f7e nodeName:}" failed. No retries permitted until 2026-07-21 19:43:35.32937849 +0000 UTC m=+378.513820152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/55620973-5262-462b-bbd4-a9f22b9d6f7e-cert") pod "llmisvc-controller-manager-777c6bd7d5-tc5xz" (UID: "55620973-5262-462b-bbd4-a9f22b9d6f7e") : secret "llmisvc-webhook-server-cert" not found Jul 21 19:43:34.829567 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:43:34.829419 2557 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 21 19:43:34.829567 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:43:34.829488 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3956c1ef-0b89-4e91-a16d-d08fa3f87476-cert podName:3956c1ef-0b89-4e91-a16d-d08fa3f87476 nodeName:}" failed. No retries permitted until 2026-07-21 19:43:35.329469891 +0000 UTC m=+378.513911555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3956c1ef-0b89-4e91-a16d-d08fa3f87476-cert") pod "kserve-controller-manager-d6f4b65b8-84kp2" (UID: "3956c1ef-0b89-4e91-a16d-d08fa3f87476") : secret "kserve-webhook-server-cert" not found Jul 21 19:43:34.846539 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.846510 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g42xz\" (UniqueName: \"kubernetes.io/projected/55620973-5262-462b-bbd4-a9f22b9d6f7e-kube-api-access-g42xz\") pod \"llmisvc-controller-manager-777c6bd7d5-tc5xz\" (UID: \"55620973-5262-462b-bbd4-a9f22b9d6f7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:34.846823 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.846797 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cnfs\" (UniqueName: \"kubernetes.io/projected/3956c1ef-0b89-4e91-a16d-d08fa3f87476-kube-api-access-9cnfs\") pod \"kserve-controller-manager-d6f4b65b8-84kp2\" (UID: \"3956c1ef-0b89-4e91-a16d-d08fa3f87476\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:34.992487 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.992454 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-shf98"] Jul 21 19:43:34.992691 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:34.992654 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" podUID="7e19609d-ee10-4caa-b77b-b55621d8f5fd" containerName="discovery" containerID="cri-o://9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2" gracePeriod=30 Jul 21 19:43:35.175702 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.175661 2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-shf98 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:8080/ready\": dial tcp 10.134.0.31:8080: connect: connection refused" start-of-body= Jul 21 19:43:35.175908 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.175724 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" podUID="7e19609d-ee10-4caa-b77b-b55621d8f5fd" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.31:8080/ready\": dial tcp 10.134.0.31:8080: connect: connection refused" Jul 21 19:43:35.334370 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.334281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3956c1ef-0b89-4e91-a16d-d08fa3f87476-cert\") pod \"kserve-controller-manager-d6f4b65b8-84kp2\" (UID: \"3956c1ef-0b89-4e91-a16d-d08fa3f87476\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:35.334542 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.334378 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55620973-5262-462b-bbd4-a9f22b9d6f7e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tc5xz\" (UID: \"55620973-5262-462b-bbd4-a9f22b9d6f7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:35.336914 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.336890 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3956c1ef-0b89-4e91-a16d-d08fa3f87476-cert\") pod \"kserve-controller-manager-d6f4b65b8-84kp2\" (UID: \"3956c1ef-0b89-4e91-a16d-d08fa3f87476\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:35.337029 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.336973 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55620973-5262-462b-bbd4-a9f22b9d6f7e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tc5xz\" (UID: \"55620973-5262-462b-bbd4-a9f22b9d6f7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:35.528448 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.528411 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:35.536479 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.536320 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:35.564026 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.563991 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:43:35.564173 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.564078 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:43:35.564173 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.564130 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:43:35.646681 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.646650 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:35.714663 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.714594 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-84kp2"] Jul 21 19:43:35.718002 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:43:35.717950 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3956c1ef_0b89_4e91_a16d_d08fa3f87476.slice/crio-0da89c20a39456f81d37de859abb4f66aed6ad4a137c0b56aa5b1c1f3f906f3a WatchSource:0}: Error finding container 0da89c20a39456f81d37de859abb4f66aed6ad4a137c0b56aa5b1c1f3f906f3a: Status 404 returned error can't find the container with id 0da89c20a39456f81d37de859abb4f66aed6ad4a137c0b56aa5b1c1f3f906f3a Jul 21 19:43:35.737853 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.737829 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-token\") pod \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " Jul 21 19:43:35.737992 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.737887 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e19609d-ee10-4caa-b77b-b55621d8f5fd-local-certs\") pod \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " Jul 21 19:43:35.737992 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.737930 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgtvj\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-kube-api-access-hgtvj\") pod \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " Jul 21 19:43:35.737992 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.737966 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-ca-configmap\") pod \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " Jul 21 19:43:35.738174 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.738054 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-kubeconfig\") pod \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " Jul 21 19:43:35.738174 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.738107 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-cacerts\") pod \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " Jul 21 19:43:35.738174 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.738157 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-dns-cert\") pod \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\" (UID: \"7e19609d-ee10-4caa-b77b-b55621d8f5fd\") " Jul 21 19:43:35.738424 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.738389 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7e19609d-ee10-4caa-b77b-b55621d8f5fd" (UID: "7e19609d-ee10-4caa-b77b-b55621d8f5fd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:43:35.738513 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.738489 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-ca-configmap\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:43:35.740852 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.740825 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7e19609d-ee10-4caa-b77b-b55621d8f5fd" (UID: "7e19609d-ee10-4caa-b77b-b55621d8f5fd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:43:35.740976 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.740846 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7e19609d-ee10-4caa-b77b-b55621d8f5fd" (UID: "7e19609d-ee10-4caa-b77b-b55621d8f5fd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:43:35.740976 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.740896 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-kube-api-access-hgtvj" (OuterVolumeSpecName: "kube-api-access-hgtvj") pod "7e19609d-ee10-4caa-b77b-b55621d8f5fd" (UID: "7e19609d-ee10-4caa-b77b-b55621d8f5fd"). InnerVolumeSpecName "kube-api-access-hgtvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:43:35.741192 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.741173 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e19609d-ee10-4caa-b77b-b55621d8f5fd-local-certs" (OuterVolumeSpecName: "local-certs") pod "7e19609d-ee10-4caa-b77b-b55621d8f5fd" (UID: "7e19609d-ee10-4caa-b77b-b55621d8f5fd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:43:35.741272 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.741224 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-cacerts" (OuterVolumeSpecName: "cacerts") pod "7e19609d-ee10-4caa-b77b-b55621d8f5fd" (UID: "7e19609d-ee10-4caa-b77b-b55621d8f5fd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:43:35.741325 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.741308 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e19609d-ee10-4caa-b77b-b55621d8f5fd" (UID: "7e19609d-ee10-4caa-b77b-b55621d8f5fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:43:35.748452 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.748430 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz"] Jul 21 19:43:35.749908 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:43:35.749869 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod55620973_5262_462b_bbd4_a9f22b9d6f7e.slice/crio-bfe49a33a7f73357f49115e82ced4eed9ad78c4d9c2252c0a30779de7c70d29b WatchSource:0}: Error finding container bfe49a33a7f73357f49115e82ced4eed9ad78c4d9c2252c0a30779de7c70d29b: Status 404 returned error can't find the container with id bfe49a33a7f73357f49115e82ced4eed9ad78c4d9c2252c0a30779de7c70d29b Jul 21 19:43:35.839615 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.839534 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgtvj\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-kube-api-access-hgtvj\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:43:35.839615 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.839562 2557 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-kubeconfig\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:43:35.839615 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.839573 2557 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-cacerts\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:43:35.839615 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.839583 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-csr-dns-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:43:35.839615 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.839593 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e19609d-ee10-4caa-b77b-b55621d8f5fd-istio-token\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:43:35.839615 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:35.839601 2557 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e19609d-ee10-4caa-b77b-b55621d8f5fd-local-certs\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:43:36.196033 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.195998 2557 generic.go:356] "Generic (PLEG): container finished" podID="7e19609d-ee10-4caa-b77b-b55621d8f5fd" containerID="9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2" exitCode=0 Jul 21 19:43:36.196248 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.196053 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" Jul 21 19:43:36.196248 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.196076 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" event={"ID":"7e19609d-ee10-4caa-b77b-b55621d8f5fd","Type":"ContainerDied","Data":"9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2"} Jul 21 19:43:36.196248 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.196120 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-shf98" event={"ID":"7e19609d-ee10-4caa-b77b-b55621d8f5fd","Type":"ContainerDied","Data":"a6df64fb7318446fa4afd18487d5b221ab6dda6b811866d19e9db8fab1ccc2fe"} Jul 21 19:43:36.196248 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.196146 2557 scope.go:117] "RemoveContainer" containerID="9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2" Jul 21 19:43:36.197594 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.197368 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" event={"ID":"3956c1ef-0b89-4e91-a16d-d08fa3f87476","Type":"ContainerStarted","Data":"0da89c20a39456f81d37de859abb4f66aed6ad4a137c0b56aa5b1c1f3f906f3a"} Jul 21 19:43:36.199080 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.199056 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" event={"ID":"2ffb7861-e586-4d4b-afe4-67165d82f0b0","Type":"ContainerStarted","Data":"38b72c75907d272c61dbafc91736037152cbc8ef3c0945b1d86ca7f8f16d0640"} Jul 21 19:43:36.200355 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.200332 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" event={"ID":"55620973-5262-462b-bbd4-a9f22b9d6f7e","Type":"ContainerStarted","Data":"bfe49a33a7f73357f49115e82ced4eed9ad78c4d9c2252c0a30779de7c70d29b"} Jul 21 19:43:36.205518 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.205504 2557 scope.go:117] "RemoveContainer" containerID="9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2" Jul 21 19:43:36.205762 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:43:36.205744 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2\": container with ID starting with 9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2 not found: ID does not exist" containerID="9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2" Jul 21 19:43:36.205806 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.205770 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2"} err="failed to get container status \"9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2\": rpc error: code = NotFound desc = could not find container \"9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2\": container with ID starting with 9f2ce411797bc1522053938098668482f7a10d793d1544b5da6948e974fb26f2 not found: ID does not exist" Jul 21 19:43:36.227133 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.227086 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" podStartSLOduration=1.218816504 podStartE2EDuration="4.227069954s" totalImagesPullingTime="3.00825345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:43:36.226311413 +0000 UTC m=+379.410753127" watchObservedRunningTime="2026-07-21 19:43:36.227069954 +0000 UTC m=+379.411511634" Jul 21 19:43:36.246046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.246016 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-shf98"] Jul 21 19:43:36.257453 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.255236 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-shf98"] Jul 21 19:43:36.384974 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.384931 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:36.391609 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.391566 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:36.505899 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:36.505798 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e19609d-ee10-4caa-b77b-b55621d8f5fd" path="/var/lib/kubelet/pods/7e19609d-ee10-4caa-b77b-b55621d8f5fd/volumes" Jul 21 19:43:37.207859 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:37.207831 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:37.208834 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:37.208805 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rrvcj" Jul 21 19:43:38.214907 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:38.214852 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" event={"ID":"3956c1ef-0b89-4e91-a16d-d08fa3f87476","Type":"ContainerStarted","Data":"fc0508011ca2ae4e18790556b4fb9c4242174025a9c8ef447f04fcbe8f4ba4ca"} Jul 21 19:43:38.215352 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:38.214952 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:43:38.235071 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:38.235023 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" podStartSLOduration=1.93961625 podStartE2EDuration="4.235006755s" totalImagesPullingTime="2.295390505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:43:38.234602944 +0000 UTC m=+381.419044644" watchObservedRunningTime="2026-07-21 19:43:38.235006755 +0000 UTC m=+381.419448435" Jul 21 19:43:39.221268 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:39.221233 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" event={"ID":"55620973-5262-462b-bbd4-a9f22b9d6f7e","Type":"ContainerStarted","Data":"bdf0c1558e146d6d3fdce4d3d9fcc1bdc332aee44650d4230b1361a02bea0f25"} Jul 21 19:43:39.221736 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:39.221572 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:43:39.243754 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:39.243703 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" podStartSLOduration=2.172075369 podStartE2EDuration="5.243687298s" totalImagesPullingTime="3.071611929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:43:39.241692748 +0000 UTC m=+382.426134447" watchObservedRunningTime="2026-07-21 19:43:39.243687298 +0000 UTC m=+382.428129008" Jul 21 19:43:40.166428 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:40.166400 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-k86fn" Jul 21 19:43:40.168960 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:43:40.168937 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-lzs2n" Jul 21 19:44:09.227109 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:09.227079 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-84kp2" Jul 21 19:44:10.228748 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:10.228717 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc5xz" Jul 21 19:44:24.053144 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.053111 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rdzw5"] Jul 21 19:44:24.053620 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.053435 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e19609d-ee10-4caa-b77b-b55621d8f5fd" containerName="discovery" Jul 21 19:44:24.053620 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.053446 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e19609d-ee10-4caa-b77b-b55621d8f5fd" containerName="discovery" Jul 21 19:44:24.053620 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.053524 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e19609d-ee10-4caa-b77b-b55621d8f5fd" containerName="discovery" Jul 21 19:44:24.056037 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.056011 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" Jul 21 19:44:24.059496 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.059470 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-269q8\"" Jul 21 19:44:24.060586 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.060563 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 19:44:24.060695 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.060590 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 19:44:24.070845 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.070822 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rdzw5"] Jul 21 19:44:24.159441 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.159403 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crl5l\" (UniqueName: \"kubernetes.io/projected/4f935f2d-2150-4644-b406-5705e75c6be0-kube-api-access-crl5l\") pod \"kuadrant-operator-catalog-rdzw5\" (UID: \"4f935f2d-2150-4644-b406-5705e75c6be0\") " pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" Jul 21 19:44:24.260287 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.260244 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crl5l\" (UniqueName: \"kubernetes.io/projected/4f935f2d-2150-4644-b406-5705e75c6be0-kube-api-access-crl5l\") pod \"kuadrant-operator-catalog-rdzw5\" (UID: \"4f935f2d-2150-4644-b406-5705e75c6be0\") " pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" Jul 21 19:44:24.272424 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.272393 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crl5l\" (UniqueName: \"kubernetes.io/projected/4f935f2d-2150-4644-b406-5705e75c6be0-kube-api-access-crl5l\") pod \"kuadrant-operator-catalog-rdzw5\" (UID: \"4f935f2d-2150-4644-b406-5705e75c6be0\") " pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" Jul 21 19:44:24.366151 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.366056 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" Jul 21 19:44:24.412892 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.412841 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rdzw5"] Jul 21 19:44:24.503765 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:44:24.503741 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f935f2d_2150_4644_b406_5705e75c6be0.slice/crio-bd27aeffcb4e6558c2562b02448fa13b43fa72980fa2cb72b5378d68a437e026 WatchSource:0}: Error finding container bd27aeffcb4e6558c2562b02448fa13b43fa72980fa2cb72b5378d68a437e026: Status 404 returned error can't find the container with id bd27aeffcb4e6558c2562b02448fa13b43fa72980fa2cb72b5378d68a437e026 Jul 21 19:44:24.505432 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.505398 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rdzw5"] Jul 21 19:44:24.614943 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.614910 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zlwf5"] Jul 21 19:44:24.617509 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.617465 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" Jul 21 19:44:24.627929 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.627907 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zlwf5"] Jul 21 19:44:24.765562 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.765530 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzk7\" (UniqueName: \"kubernetes.io/projected/bbdca730-24fc-411b-9c6a-a5b8a09a9ded-kube-api-access-kkzk7\") pod \"kuadrant-operator-catalog-zlwf5\" (UID: \"bbdca730-24fc-411b-9c6a-a5b8a09a9ded\") " pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" Jul 21 19:44:24.866339 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.866306 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzk7\" (UniqueName: \"kubernetes.io/projected/bbdca730-24fc-411b-9c6a-a5b8a09a9ded-kube-api-access-kkzk7\") pod \"kuadrant-operator-catalog-zlwf5\" (UID: \"bbdca730-24fc-411b-9c6a-a5b8a09a9ded\") " pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" Jul 21 19:44:24.875355 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.875285 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzk7\" (UniqueName: \"kubernetes.io/projected/bbdca730-24fc-411b-9c6a-a5b8a09a9ded-kube-api-access-kkzk7\") pod \"kuadrant-operator-catalog-zlwf5\" (UID: \"bbdca730-24fc-411b-9c6a-a5b8a09a9ded\") " pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" Jul 21 19:44:24.926890 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:24.926848 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" Jul 21 19:44:25.074647 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:25.074620 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zlwf5"] Jul 21 19:44:25.102575 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:44:25.102545 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdca730_24fc_411b_9c6a_a5b8a09a9ded.slice/crio-352333e7a380e3c77bf8231fc0c904659f12822ddac5835f7166448b3bf0b0c9 WatchSource:0}: Error finding container 352333e7a380e3c77bf8231fc0c904659f12822ddac5835f7166448b3bf0b0c9: Status 404 returned error can't find the container with id 352333e7a380e3c77bf8231fc0c904659f12822ddac5835f7166448b3bf0b0c9 Jul 21 19:44:25.436749 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:25.436714 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" event={"ID":"bbdca730-24fc-411b-9c6a-a5b8a09a9ded","Type":"ContainerStarted","Data":"352333e7a380e3c77bf8231fc0c904659f12822ddac5835f7166448b3bf0b0c9"} Jul 21 19:44:25.438072 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:25.438009 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" event={"ID":"4f935f2d-2150-4644-b406-5705e75c6be0","Type":"ContainerStarted","Data":"bd27aeffcb4e6558c2562b02448fa13b43fa72980fa2cb72b5378d68a437e026"} Jul 21 19:44:27.448967 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:27.448929 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" event={"ID":"bbdca730-24fc-411b-9c6a-a5b8a09a9ded","Type":"ContainerStarted","Data":"e7fb7a141afe0acf761dc0cd2ce780eed41734df214b16180f548da89cdc7df7"} Jul 21 19:44:27.450295 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:27.450268 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" event={"ID":"4f935f2d-2150-4644-b406-5705e75c6be0","Type":"ContainerStarted","Data":"f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f"} Jul 21 19:44:27.450439 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:27.450358 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" podUID="4f935f2d-2150-4644-b406-5705e75c6be0" containerName="registry-server" containerID="cri-o://f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f" gracePeriod=2 Jul 21 19:44:27.468838 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:27.468793 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" podStartSLOduration=2.084331583 podStartE2EDuration="3.468778203s" totalImagesPullingTime="1.38444662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:44:27.466696074 +0000 UTC m=+430.651137782" watchObservedRunningTime="2026-07-21 19:44:27.468778203 +0000 UTC m=+430.653219904" Jul 21 19:44:27.485685 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:27.485641 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" podStartSLOduration=1.505111081 podStartE2EDuration="3.485625942s" totalImagesPullingTime="1.980514861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:44:27.484991272 +0000 UTC m=+430.669432986" watchObservedRunningTime="2026-07-21 19:44:27.485625942 +0000 UTC m=+430.670067622" Jul 21 19:44:27.701509 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:27.701454 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" Jul 21 19:44:27.797099 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:27.797067 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crl5l\" (UniqueName: \"kubernetes.io/projected/4f935f2d-2150-4644-b406-5705e75c6be0-kube-api-access-crl5l\") pod \"4f935f2d-2150-4644-b406-5705e75c6be0\" (UID: \"4f935f2d-2150-4644-b406-5705e75c6be0\") " Jul 21 19:44:27.799249 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:27.799226 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f935f2d-2150-4644-b406-5705e75c6be0-kube-api-access-crl5l" (OuterVolumeSpecName: "kube-api-access-crl5l") pod "4f935f2d-2150-4644-b406-5705e75c6be0" (UID: "4f935f2d-2150-4644-b406-5705e75c6be0"). InnerVolumeSpecName "kube-api-access-crl5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:44:27.898629 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:27.898592 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crl5l\" (UniqueName: \"kubernetes.io/projected/4f935f2d-2150-4644-b406-5705e75c6be0-kube-api-access-crl5l\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:28.455623 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.455584 2557 generic.go:356] "Generic (PLEG): container finished" podID="4f935f2d-2150-4644-b406-5705e75c6be0" containerID="f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f" exitCode=0 Jul 21 19:44:28.456051 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.455632 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" Jul 21 19:44:28.456051 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.455666 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" event={"ID":"4f935f2d-2150-4644-b406-5705e75c6be0","Type":"ContainerDied","Data":"f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f"} Jul 21 19:44:28.456051 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.455713 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rdzw5" event={"ID":"4f935f2d-2150-4644-b406-5705e75c6be0","Type":"ContainerDied","Data":"bd27aeffcb4e6558c2562b02448fa13b43fa72980fa2cb72b5378d68a437e026"} Jul 21 19:44:28.456051 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.455732 2557 scope.go:117] "RemoveContainer" containerID="f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f" Jul 21 19:44:28.464819 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.464798 2557 scope.go:117] "RemoveContainer" containerID="f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f" Jul 21 19:44:28.465088 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:44:28.465069 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f\": container with ID starting with f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f not found: ID does not exist" containerID="f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f" Jul 21 19:44:28.465170 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.465102 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f"} err="failed to get container status \"f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f\": rpc error: code = NotFound desc = could not find container \"f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f\": container with ID starting with f22010f79dfff85c6c146772c0ea12a5632138ab659e799ce9b4eceb1b586d8f not found: ID does not exist" Jul 21 19:44:28.477805 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.477781 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rdzw5"] Jul 21 19:44:28.484305 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.484284 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rdzw5"] Jul 21 19:44:28.503955 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:28.503932 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f935f2d-2150-4644-b406-5705e75c6be0" path="/var/lib/kubelet/pods/4f935f2d-2150-4644-b406-5705e75c6be0/volumes" Jul 21 19:44:34.927540 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:34.927487 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" Jul 21 19:44:34.927995 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:34.927565 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" Jul 21 19:44:34.950091 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:34.950057 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" Jul 21 19:44:35.510548 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:35.510520 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zlwf5" Jul 21 19:44:36.401357 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.401323 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-99458d98-7mkt2"] Jul 21 19:44:36.401831 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.401811 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f935f2d-2150-4644-b406-5705e75c6be0" containerName="registry-server" Jul 21 19:44:36.401916 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.401837 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f935f2d-2150-4644-b406-5705e75c6be0" containerName="registry-server" Jul 21 19:44:36.401987 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.401974 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f935f2d-2150-4644-b406-5705e75c6be0" containerName="registry-server" Jul 21 19:44:36.406687 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.406664 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.417911 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.417869 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99458d98-7mkt2"] Jul 21 19:44:36.473441 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.473409 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-console-config\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.473597 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.473473 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ec39da6-4a11-4bae-8264-bbbf43286441-console-serving-cert\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.473597 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.473492 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-oauth-serving-cert\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.473597 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.473513 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-service-ca\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.473597 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.473529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-trusted-ca-bundle\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.473597 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.473548 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plghc\" (UniqueName: \"kubernetes.io/projected/0ec39da6-4a11-4bae-8264-bbbf43286441-kube-api-access-plghc\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.473597 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.473597 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ec39da6-4a11-4bae-8264-bbbf43286441-console-oauth-config\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.574936 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.574841 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ec39da6-4a11-4bae-8264-bbbf43286441-console-oauth-config\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.575110 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.575005 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-console-config\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.575110 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.575104 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ec39da6-4a11-4bae-8264-bbbf43286441-console-serving-cert\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.575364 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.575331 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-oauth-serving-cert\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.575453 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.575384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-service-ca\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.575453 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.575407 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-trusted-ca-bundle\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.575453 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.575441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plghc\" (UniqueName: \"kubernetes.io/projected/0ec39da6-4a11-4bae-8264-bbbf43286441-kube-api-access-plghc\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.575758 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.575730 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-console-config\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.576035 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.576016 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-service-ca\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.576132 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.576107 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-oauth-serving-cert\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.576280 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.576262 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ec39da6-4a11-4bae-8264-bbbf43286441-trusted-ca-bundle\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.577414 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.577381 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ec39da6-4a11-4bae-8264-bbbf43286441-console-serving-cert\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.577522 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.577423 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ec39da6-4a11-4bae-8264-bbbf43286441-console-oauth-config\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.583994 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.583974 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plghc\" (UniqueName: \"kubernetes.io/projected/0ec39da6-4a11-4bae-8264-bbbf43286441-kube-api-access-plghc\") pod \"console-99458d98-7mkt2\" (UID: \"0ec39da6-4a11-4bae-8264-bbbf43286441\") " pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.716769 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.716664 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:36.855521 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:36.855493 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99458d98-7mkt2"] Jul 21 19:44:36.857310 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:44:36.857281 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec39da6_4a11_4bae_8264_bbbf43286441.slice/crio-7e1fa382a343102ab95222047301462faa7fb72789aaf418ff9a7297ed6880d9 WatchSource:0}: Error finding container 7e1fa382a343102ab95222047301462faa7fb72789aaf418ff9a7297ed6880d9: Status 404 returned error can't find the container with id 7e1fa382a343102ab95222047301462faa7fb72789aaf418ff9a7297ed6880d9 Jul 21 19:44:37.505384 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:37.505347 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99458d98-7mkt2" event={"ID":"0ec39da6-4a11-4bae-8264-bbbf43286441","Type":"ContainerStarted","Data":"5b1fbbf5c9aecef47c223bec818f20690722f6444bc87db68a1145c0f2d7b090"} Jul 21 19:44:37.505777 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:37.505425 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99458d98-7mkt2" event={"ID":"0ec39da6-4a11-4bae-8264-bbbf43286441","Type":"ContainerStarted","Data":"7e1fa382a343102ab95222047301462faa7fb72789aaf418ff9a7297ed6880d9"} Jul 21 19:44:37.527672 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:37.527618 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-99458d98-7mkt2" podStartSLOduration=1.527602249 podStartE2EDuration="1.527602249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:44:37.5259291 +0000 UTC m=+440.710370802" watchObservedRunningTime="2026-07-21 19:44:37.527602249 +0000 UTC m=+440.712043930" Jul 21 19:44:40.033265 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.033184 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb"] Jul 21 19:44:40.035961 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.035944 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.038819 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.038802 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zcmqw\"" Jul 21 19:44:40.045357 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.045333 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb"] Jul 21 19:44:40.105523 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.105488 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.105736 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.105547 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.105736 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.105599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5qng\" (UniqueName: \"kubernetes.io/projected/60086618-c52f-4e93-9514-02446c4bed24-kube-api-access-n5qng\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.206636 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.206597 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5qng\" (UniqueName: \"kubernetes.io/projected/60086618-c52f-4e93-9514-02446c4bed24-kube-api-access-n5qng\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.206803 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.206675 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.206803 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.206717 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.207105 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.207085 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.207157 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.207107 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.216029 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.216004 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5qng\" (UniqueName: \"kubernetes.io/projected/60086618-c52f-4e93-9514-02446c4bed24-kube-api-access-n5qng\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.346412 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.346332 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:40.480579 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.480553 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb"] Jul 21 19:44:40.482276 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:44:40.482249 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60086618_c52f_4e93_9514_02446c4bed24.slice/crio-05681d5a9fc07d701b759a2a323bafec30463ed738c1b3980de9a980d116a3d7 WatchSource:0}: Error finding container 05681d5a9fc07d701b759a2a323bafec30463ed738c1b3980de9a980d116a3d7: Status 404 returned error can't find the container with id 05681d5a9fc07d701b759a2a323bafec30463ed738c1b3980de9a980d116a3d7 Jul 21 19:44:40.519269 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.519223 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" event={"ID":"60086618-c52f-4e93-9514-02446c4bed24","Type":"ContainerStarted","Data":"05681d5a9fc07d701b759a2a323bafec30463ed738c1b3980de9a980d116a3d7"} Jul 21 19:44:40.831713 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.831673 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx"] Jul 21 19:44:40.835146 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.835124 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:40.844423 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.844398 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx"] Jul 21 19:44:40.912484 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.912451 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:40.912679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.912506 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qfk\" (UniqueName: \"kubernetes.io/projected/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-kube-api-access-h5qfk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:40.912679 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:40.912597 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:41.013671 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.013634 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:41.013828 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.013687 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qfk\" (UniqueName: \"kubernetes.io/projected/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-kube-api-access-h5qfk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:41.013828 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.013708 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:41.014051 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.014031 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:41.014091 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.014062 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:41.023201 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.023177 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qfk\" (UniqueName: \"kubernetes.io/projected/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-kube-api-access-h5qfk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:41.144490 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.144461 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:41.279149 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.279124 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx"] Jul 21 19:44:41.281109 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:44:41.281072 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d7fdbb_586f_4147_bcc9_68cd83a5bae9.slice/crio-064da45b5281656c7189951216f28303c6e7c8629ab9a78ca8d5fe5e6e620093 WatchSource:0}: Error finding container 064da45b5281656c7189951216f28303c6e7c8629ab9a78ca8d5fe5e6e620093: Status 404 returned error can't find the container with id 064da45b5281656c7189951216f28303c6e7c8629ab9a78ca8d5fe5e6e620093 Jul 21 19:44:41.445178 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.445142 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn"] Jul 21 19:44:41.448732 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.448685 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.457558 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.457530 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn"] Jul 21 19:44:41.518320 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.518285 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.518477 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.518342 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6kfl\" (UniqueName: \"kubernetes.io/projected/edefb881-6ed1-478a-ba17-ee65c1886005-kube-api-access-p6kfl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.518477 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.518454 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.525603 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.525579 2557 generic.go:356] "Generic (PLEG): container finished" podID="60086618-c52f-4e93-9514-02446c4bed24" containerID="9a35a7f2ad5025db85670454489e6d6b109d02aa2261ba67854d39577c25c4ab" exitCode=0 Jul 21 19:44:41.525741 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.525672 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" event={"ID":"60086618-c52f-4e93-9514-02446c4bed24","Type":"ContainerDied","Data":"9a35a7f2ad5025db85670454489e6d6b109d02aa2261ba67854d39577c25c4ab"} Jul 21 19:44:41.527211 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.527191 2557 generic.go:356] "Generic (PLEG): container finished" podID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerID="7648280c19725df345e334b2d6b53a784300278f9368af8f50fbb32e2659a3e4" exitCode=0 Jul 21 19:44:41.527288 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.527272 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" event={"ID":"27d7fdbb-586f-4147-bcc9-68cd83a5bae9","Type":"ContainerDied","Data":"7648280c19725df345e334b2d6b53a784300278f9368af8f50fbb32e2659a3e4"} Jul 21 19:44:41.527334 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.527308 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" event={"ID":"27d7fdbb-586f-4147-bcc9-68cd83a5bae9","Type":"ContainerStarted","Data":"064da45b5281656c7189951216f28303c6e7c8629ab9a78ca8d5fe5e6e620093"} Jul 21 19:44:41.619379 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.619339 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.619548 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.619397 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.619548 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.619448 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6kfl\" (UniqueName: \"kubernetes.io/projected/edefb881-6ed1-478a-ba17-ee65c1886005-kube-api-access-p6kfl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.619833 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.619808 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.619869 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.619824 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.632944 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.632914 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6kfl\" (UniqueName: \"kubernetes.io/projected/edefb881-6ed1-478a-ba17-ee65c1886005-kube-api-access-p6kfl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.758368 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.758277 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:41.826537 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.826509 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck"] Jul 21 19:44:41.830312 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.830292 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:41.839395 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.839369 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck"] Jul 21 19:44:41.895628 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.895601 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn"] Jul 21 19:44:41.897219 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:44:41.897194 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedefb881_6ed1_478a_ba17_ee65c1886005.slice/crio-a29bcbd5ad10071dc98d2aa36e6f59a948f82e3f3321b05604097a3e4181cded WatchSource:0}: Error finding container a29bcbd5ad10071dc98d2aa36e6f59a948f82e3f3321b05604097a3e4181cded: Status 404 returned error can't find the container with id a29bcbd5ad10071dc98d2aa36e6f59a948f82e3f3321b05604097a3e4181cded Jul 21 19:44:41.922270 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.922242 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:41.922405 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.922283 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvct\" (UniqueName: \"kubernetes.io/projected/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-kube-api-access-bxvct\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:41.922405 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:41.922355 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:42.023611 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.023552 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:42.023611 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.023592 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvct\" (UniqueName: \"kubernetes.io/projected/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-kube-api-access-bxvct\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:42.023749 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.023625 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:42.023988 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.023965 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:42.024030 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.024008 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:42.032919 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.032893 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvct\" (UniqueName: \"kubernetes.io/projected/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-kube-api-access-bxvct\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:42.141569 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.141534 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:42.396027 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.396005 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck"] Jul 21 19:44:42.397810 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:44:42.397782 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a97611e_96c7_4b3d_a164_d9e8ff1cae22.slice/crio-710f3cc4c15618b8831703c9a21d66914986ea20de2a59e1b82380782c8cb818 WatchSource:0}: Error finding container 710f3cc4c15618b8831703c9a21d66914986ea20de2a59e1b82380782c8cb818: Status 404 returned error can't find the container with id 710f3cc4c15618b8831703c9a21d66914986ea20de2a59e1b82380782c8cb818 Jul 21 19:44:42.533202 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.533170 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" event={"ID":"27d7fdbb-586f-4147-bcc9-68cd83a5bae9","Type":"ContainerStarted","Data":"d9dd864a509d641a216365370751e2f2db66f074a3958e870b01ffedfb76447a"} Jul 21 19:44:42.534555 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.534531 2557 generic.go:356] "Generic (PLEG): container finished" podID="edefb881-6ed1-478a-ba17-ee65c1886005" containerID="5999f6badf488b821940cbe198f3ad80800c130b89a4818fb3a49db318611a37" exitCode=0 Jul 21 19:44:42.534674 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.534615 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" event={"ID":"edefb881-6ed1-478a-ba17-ee65c1886005","Type":"ContainerDied","Data":"5999f6badf488b821940cbe198f3ad80800c130b89a4818fb3a49db318611a37"} Jul 21 19:44:42.534674 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.534663 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" event={"ID":"edefb881-6ed1-478a-ba17-ee65c1886005","Type":"ContainerStarted","Data":"a29bcbd5ad10071dc98d2aa36e6f59a948f82e3f3321b05604097a3e4181cded"} Jul 21 19:44:42.536132 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.536110 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" event={"ID":"7a97611e-96c7-4b3d-a164-d9e8ff1cae22","Type":"ContainerStarted","Data":"54621751e6ea58f37b15f1e6b95dbcb66088d2cb74d53249ee582ce54a9c7e85"} Jul 21 19:44:42.536388 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:42.536143 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" event={"ID":"7a97611e-96c7-4b3d-a164-d9e8ff1cae22","Type":"ContainerStarted","Data":"710f3cc4c15618b8831703c9a21d66914986ea20de2a59e1b82380782c8cb818"} Jul 21 19:44:43.542665 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:43.542633 2557 generic.go:356] "Generic (PLEG): container finished" podID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerID="d9dd864a509d641a216365370751e2f2db66f074a3958e870b01ffedfb76447a" exitCode=0 Jul 21 19:44:43.543129 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:43.542713 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" event={"ID":"27d7fdbb-586f-4147-bcc9-68cd83a5bae9","Type":"ContainerDied","Data":"d9dd864a509d641a216365370751e2f2db66f074a3958e870b01ffedfb76447a"} Jul 21 19:44:43.544630 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:43.544607 2557 generic.go:356] "Generic (PLEG): container finished" podID="edefb881-6ed1-478a-ba17-ee65c1886005" containerID="ffa0b588a609914d870b9fdb0bbc649f9257bd79d1461bcf7543f765510d1b77" exitCode=0 Jul 21 19:44:43.544868 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:43.544682 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" event={"ID":"edefb881-6ed1-478a-ba17-ee65c1886005","Type":"ContainerDied","Data":"ffa0b588a609914d870b9fdb0bbc649f9257bd79d1461bcf7543f765510d1b77"} Jul 21 19:44:43.546765 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:43.546747 2557 generic.go:356] "Generic (PLEG): container finished" podID="60086618-c52f-4e93-9514-02446c4bed24" containerID="25bf4b72381350ea7bf2692bf6aaea98c14722320401d2d448c9d876e8539d4d" exitCode=0 Jul 21 19:44:43.546870 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:43.546835 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" event={"ID":"60086618-c52f-4e93-9514-02446c4bed24","Type":"ContainerDied","Data":"25bf4b72381350ea7bf2692bf6aaea98c14722320401d2d448c9d876e8539d4d"} Jul 21 19:44:43.548574 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:43.548552 2557 generic.go:356] "Generic (PLEG): container finished" podID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerID="54621751e6ea58f37b15f1e6b95dbcb66088d2cb74d53249ee582ce54a9c7e85" exitCode=0 Jul 21 19:44:43.548663 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:43.548590 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" event={"ID":"7a97611e-96c7-4b3d-a164-d9e8ff1cae22","Type":"ContainerDied","Data":"54621751e6ea58f37b15f1e6b95dbcb66088d2cb74d53249ee582ce54a9c7e85"} Jul 21 19:44:44.555556 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:44.555520 2557 generic.go:356] "Generic (PLEG): container finished" podID="60086618-c52f-4e93-9514-02446c4bed24" containerID="9a224b4397666cbdd64dc232cbfcf288caebbcccd3b03b874fd6c4248832a7e5" exitCode=0 Jul 21 19:44:44.555961 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:44.555611 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" event={"ID":"60086618-c52f-4e93-9514-02446c4bed24","Type":"ContainerDied","Data":"9a224b4397666cbdd64dc232cbfcf288caebbcccd3b03b874fd6c4248832a7e5"} Jul 21 19:44:44.557265 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:44.557239 2557 generic.go:356] "Generic (PLEG): container finished" podID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerID="8ee0218a8b0cbc1c0219240a1844428fb4571b26979f59e71d51474e09c946e7" exitCode=0 Jul 21 19:44:44.557365 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:44.557326 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" event={"ID":"7a97611e-96c7-4b3d-a164-d9e8ff1cae22","Type":"ContainerDied","Data":"8ee0218a8b0cbc1c0219240a1844428fb4571b26979f59e71d51474e09c946e7"} Jul 21 19:44:44.559521 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:44.559497 2557 generic.go:356] "Generic (PLEG): container finished" podID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerID="15551007e9dd6ef272bc396ff9ecf6dbddc943fdfc7c243d355d89932e36eabc" exitCode=0 Jul 21 19:44:44.559616 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:44.559595 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" event={"ID":"27d7fdbb-586f-4147-bcc9-68cd83a5bae9","Type":"ContainerDied","Data":"15551007e9dd6ef272bc396ff9ecf6dbddc943fdfc7c243d355d89932e36eabc"} Jul 21 19:44:44.561618 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:44.561597 2557 generic.go:356] "Generic (PLEG): container finished" podID="edefb881-6ed1-478a-ba17-ee65c1886005" containerID="6afce846135c3f44317f1503bb1c620dc3210e652a789ff886768a20615e75a3" exitCode=0 Jul 21 19:44:44.561696 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:44.561671 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" event={"ID":"edefb881-6ed1-478a-ba17-ee65c1886005","Type":"ContainerDied","Data":"6afce846135c3f44317f1503bb1c620dc3210e652a789ff886768a20615e75a3"} Jul 21 19:44:45.568745 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.568714 2557 generic.go:356] "Generic (PLEG): container finished" podID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerID="57913bbf7d41e8c0d14f257eeb850aeee4a1c864bbd18a70f092d1701d932c41" exitCode=0 Jul 21 19:44:45.569175 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.568787 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" event={"ID":"7a97611e-96c7-4b3d-a164-d9e8ff1cae22","Type":"ContainerDied","Data":"57913bbf7d41e8c0d14f257eeb850aeee4a1c864bbd18a70f092d1701d932c41"} Jul 21 19:44:45.715089 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.715065 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:45.757111 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.757083 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-util\") pod \"edefb881-6ed1-478a-ba17-ee65c1886005\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " Jul 21 19:44:45.757201 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.757153 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6kfl\" (UniqueName: \"kubernetes.io/projected/edefb881-6ed1-478a-ba17-ee65c1886005-kube-api-access-p6kfl\") pod \"edefb881-6ed1-478a-ba17-ee65c1886005\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " Jul 21 19:44:45.757201 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.757190 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-bundle\") pod \"edefb881-6ed1-478a-ba17-ee65c1886005\" (UID: \"edefb881-6ed1-478a-ba17-ee65c1886005\") " Jul 21 19:44:45.757687 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.757664 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-bundle" (OuterVolumeSpecName: "bundle") pod "edefb881-6ed1-478a-ba17-ee65c1886005" (UID: "edefb881-6ed1-478a-ba17-ee65c1886005"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:44:45.759529 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.759502 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edefb881-6ed1-478a-ba17-ee65c1886005-kube-api-access-p6kfl" (OuterVolumeSpecName: "kube-api-access-p6kfl") pod "edefb881-6ed1-478a-ba17-ee65c1886005" (UID: "edefb881-6ed1-478a-ba17-ee65c1886005"). InnerVolumeSpecName "kube-api-access-p6kfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:44:45.762673 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.762650 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-util" (OuterVolumeSpecName: "util") pod "edefb881-6ed1-478a-ba17-ee65c1886005" (UID: "edefb881-6ed1-478a-ba17-ee65c1886005"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:44:45.772039 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.772019 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:45.775162 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.775144 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:45.858652 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.858561 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-bundle\") pod \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " Jul 21 19:44:45.858652 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.858612 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-util\") pod \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " Jul 21 19:44:45.858652 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.858651 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-bundle\") pod \"60086618-c52f-4e93-9514-02446c4bed24\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " Jul 21 19:44:45.858949 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.858679 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-util\") pod \"60086618-c52f-4e93-9514-02446c4bed24\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " Jul 21 19:44:45.858949 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.858706 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5qng\" (UniqueName: \"kubernetes.io/projected/60086618-c52f-4e93-9514-02446c4bed24-kube-api-access-n5qng\") pod \"60086618-c52f-4e93-9514-02446c4bed24\" (UID: \"60086618-c52f-4e93-9514-02446c4bed24\") " Jul 21 19:44:45.858949 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.858737 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5qfk\" (UniqueName: \"kubernetes.io/projected/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-kube-api-access-h5qfk\") pod \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\" (UID: \"27d7fdbb-586f-4147-bcc9-68cd83a5bae9\") " Jul 21 19:44:45.859103 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.859025 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-util\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:45.859103 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.859045 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6kfl\" (UniqueName: \"kubernetes.io/projected/edefb881-6ed1-478a-ba17-ee65c1886005-kube-api-access-p6kfl\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:45.859103 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.859058 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edefb881-6ed1-478a-ba17-ee65c1886005-bundle\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:45.859255 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.859230 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-bundle" (OuterVolumeSpecName: "bundle") pod "60086618-c52f-4e93-9514-02446c4bed24" (UID: "60086618-c52f-4e93-9514-02446c4bed24"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:44:45.859317 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.859290 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-bundle" (OuterVolumeSpecName: "bundle") pod "27d7fdbb-586f-4147-bcc9-68cd83a5bae9" (UID: "27d7fdbb-586f-4147-bcc9-68cd83a5bae9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:44:45.860944 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.860916 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-kube-api-access-h5qfk" (OuterVolumeSpecName: "kube-api-access-h5qfk") pod "27d7fdbb-586f-4147-bcc9-68cd83a5bae9" (UID: "27d7fdbb-586f-4147-bcc9-68cd83a5bae9"). InnerVolumeSpecName "kube-api-access-h5qfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:44:45.861203 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.861184 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60086618-c52f-4e93-9514-02446c4bed24-kube-api-access-n5qng" (OuterVolumeSpecName: "kube-api-access-n5qng") pod "60086618-c52f-4e93-9514-02446c4bed24" (UID: "60086618-c52f-4e93-9514-02446c4bed24"). InnerVolumeSpecName "kube-api-access-n5qng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:44:45.864616 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.864585 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-util" (OuterVolumeSpecName: "util") pod "27d7fdbb-586f-4147-bcc9-68cd83a5bae9" (UID: "27d7fdbb-586f-4147-bcc9-68cd83a5bae9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:44:45.865375 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.865355 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-util" (OuterVolumeSpecName: "util") pod "60086618-c52f-4e93-9514-02446c4bed24" (UID: "60086618-c52f-4e93-9514-02446c4bed24"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:44:45.959809 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.959773 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-bundle\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:45.959809 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.959800 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60086618-c52f-4e93-9514-02446c4bed24-util\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:45.959809 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.959812 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5qng\" (UniqueName: \"kubernetes.io/projected/60086618-c52f-4e93-9514-02446c4bed24-kube-api-access-n5qng\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:45.960055 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.959825 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5qfk\" (UniqueName: \"kubernetes.io/projected/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-kube-api-access-h5qfk\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:45.960055 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.959838 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-bundle\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:45.960055 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:45.959848 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d7fdbb-586f-4147-bcc9-68cd83a5bae9-util\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:46.576062 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.576019 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" event={"ID":"60086618-c52f-4e93-9514-02446c4bed24","Type":"ContainerDied","Data":"05681d5a9fc07d701b759a2a323bafec30463ed738c1b3980de9a980d116a3d7"} Jul 21 19:44:46.576475 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.576077 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05681d5a9fc07d701b759a2a323bafec30463ed738c1b3980de9a980d116a3d7" Jul 21 19:44:46.576475 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.576035 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb" Jul 21 19:44:46.578043 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.578003 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" Jul 21 19:44:46.578043 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.578012 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx" event={"ID":"27d7fdbb-586f-4147-bcc9-68cd83a5bae9","Type":"ContainerDied","Data":"064da45b5281656c7189951216f28303c6e7c8629ab9a78ca8d5fe5e6e620093"} Jul 21 19:44:46.578219 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.578073 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="064da45b5281656c7189951216f28303c6e7c8629ab9a78ca8d5fe5e6e620093" Jul 21 19:44:46.580213 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.580181 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" Jul 21 19:44:46.580213 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.580182 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn" event={"ID":"edefb881-6ed1-478a-ba17-ee65c1886005","Type":"ContainerDied","Data":"a29bcbd5ad10071dc98d2aa36e6f59a948f82e3f3321b05604097a3e4181cded"} Jul 21 19:44:46.580351 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.580238 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a29bcbd5ad10071dc98d2aa36e6f59a948f82e3f3321b05604097a3e4181cded" Jul 21 19:44:46.693415 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.693389 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:46.717054 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.717024 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:46.717175 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.717078 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:46.722451 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.722428 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:46.766590 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.766500 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-util\") pod \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " Jul 21 19:44:46.766590 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.766575 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxvct\" (UniqueName: \"kubernetes.io/projected/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-kube-api-access-bxvct\") pod \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " Jul 21 19:44:46.766828 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.766616 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-bundle\") pod \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\" (UID: \"7a97611e-96c7-4b3d-a164-d9e8ff1cae22\") " Jul 21 19:44:46.767491 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.767336 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-bundle" (OuterVolumeSpecName: "bundle") pod "7a97611e-96c7-4b3d-a164-d9e8ff1cae22" (UID: "7a97611e-96c7-4b3d-a164-d9e8ff1cae22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:44:46.769684 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.769651 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-kube-api-access-bxvct" (OuterVolumeSpecName: "kube-api-access-bxvct") pod "7a97611e-96c7-4b3d-a164-d9e8ff1cae22" (UID: "7a97611e-96c7-4b3d-a164-d9e8ff1cae22"). InnerVolumeSpecName "kube-api-access-bxvct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:44:46.773057 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.773036 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-util" (OuterVolumeSpecName: "util") pod "7a97611e-96c7-4b3d-a164-d9e8ff1cae22" (UID: "7a97611e-96c7-4b3d-a164-d9e8ff1cae22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:44:46.867720 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.867687 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-util\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:46.867896 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.867731 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvct\" (UniqueName: \"kubernetes.io/projected/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-kube-api-access-bxvct\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:46.867896 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:46.867741 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97611e-96c7-4b3d-a164-d9e8ff1cae22-bundle\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:44:47.587104 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:47.587072 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" Jul 21 19:44:47.587104 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:47.587095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck" event={"ID":"7a97611e-96c7-4b3d-a164-d9e8ff1cae22","Type":"ContainerDied","Data":"710f3cc4c15618b8831703c9a21d66914986ea20de2a59e1b82380782c8cb818"} Jul 21 19:44:47.587524 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:47.587140 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710f3cc4c15618b8831703c9a21d66914986ea20de2a59e1b82380782c8cb818" Jul 21 19:44:47.591429 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:47.591405 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-99458d98-7mkt2" Jul 21 19:44:47.642748 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:47.642720 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-788fc66c5d-dmt65"] Jul 21 19:44:59.352213 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352177 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm"] Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352505 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60086618-c52f-4e93-9514-02446c4bed24" containerName="util" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352518 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="60086618-c52f-4e93-9514-02446c4bed24" containerName="util" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352526 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerName="extract" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352532 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerName="extract" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352543 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60086618-c52f-4e93-9514-02446c4bed24" containerName="pull" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352548 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="60086618-c52f-4e93-9514-02446c4bed24" containerName="pull" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352556 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerName="util" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352562 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerName="util" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352568 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerName="pull" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352573 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerName="pull" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352580 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerName="util" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352584 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerName="util" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352590 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60086618-c52f-4e93-9514-02446c4bed24" containerName="extract" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352596 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="60086618-c52f-4e93-9514-02446c4bed24" containerName="extract" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352605 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edefb881-6ed1-478a-ba17-ee65c1886005" containerName="pull" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352610 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="edefb881-6ed1-478a-ba17-ee65c1886005" containerName="pull" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352616 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerName="extract" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352623 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerName="extract" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352633 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edefb881-6ed1-478a-ba17-ee65c1886005" containerName="extract" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352637 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="edefb881-6ed1-478a-ba17-ee65c1886005" containerName="extract" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352644 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edefb881-6ed1-478a-ba17-ee65c1886005" containerName="util" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352649 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="edefb881-6ed1-478a-ba17-ee65c1886005" containerName="util" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352657 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerName="pull" Jul 21 19:44:59.352659 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352661 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerName="pull" Jul 21 19:44:59.353480 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352714 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a97611e-96c7-4b3d-a164-d9e8ff1cae22" containerName="extract" Jul 21 19:44:59.353480 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352723 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d7fdbb-586f-4147-bcc9-68cd83a5bae9" containerName="extract" Jul 21 19:44:59.353480 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352729 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="60086618-c52f-4e93-9514-02446c4bed24" containerName="extract" Jul 21 19:44:59.353480 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.352736 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="edefb881-6ed1-478a-ba17-ee65c1886005" containerName="extract" Jul 21 19:44:59.361109 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.361091 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" Jul 21 19:44:59.364382 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.364357 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 19:44:59.364717 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.364695 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qlvhv\"" Jul 21 19:44:59.371953 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.371928 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm"] Jul 21 19:44:59.480993 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.480951 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxp4\" (UniqueName: \"kubernetes.io/projected/76cf9d8e-f047-49eb-a030-858aa3ddb7ef-kube-api-access-lxxp4\") pod \"dns-operator-controller-manager-864dd78c4c-bx9bm\" (UID: \"76cf9d8e-f047-49eb-a030-858aa3ddb7ef\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" Jul 21 19:44:59.582254 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.582213 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxp4\" (UniqueName: \"kubernetes.io/projected/76cf9d8e-f047-49eb-a030-858aa3ddb7ef-kube-api-access-lxxp4\") pod \"dns-operator-controller-manager-864dd78c4c-bx9bm\" (UID: \"76cf9d8e-f047-49eb-a030-858aa3ddb7ef\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" Jul 21 19:44:59.606524 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.606453 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxp4\" (UniqueName: \"kubernetes.io/projected/76cf9d8e-f047-49eb-a030-858aa3ddb7ef-kube-api-access-lxxp4\") pod \"dns-operator-controller-manager-864dd78c4c-bx9bm\" (UID: \"76cf9d8e-f047-49eb-a030-858aa3ddb7ef\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" Jul 21 19:44:59.671704 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.671672 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" Jul 21 19:44:59.810740 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:44:59.810712 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm"] Jul 21 19:44:59.812214 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:44:59.812188 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cf9d8e_f047_49eb_a030_858aa3ddb7ef.slice/crio-07d3b41a2aca11df4a5c936ff844b4701d508669b9b215e8a60b8b9c5d7b10eb WatchSource:0}: Error finding container 07d3b41a2aca11df4a5c936ff844b4701d508669b9b215e8a60b8b9c5d7b10eb: Status 404 returned error can't find the container with id 07d3b41a2aca11df4a5c936ff844b4701d508669b9b215e8a60b8b9c5d7b10eb Jul 21 19:45:00.653457 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:00.653409 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" event={"ID":"76cf9d8e-f047-49eb-a030-858aa3ddb7ef","Type":"ContainerStarted","Data":"07d3b41a2aca11df4a5c936ff844b4701d508669b9b215e8a60b8b9c5d7b10eb"} Jul 21 19:45:02.665606 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:02.665572 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" event={"ID":"76cf9d8e-f047-49eb-a030-858aa3ddb7ef","Type":"ContainerStarted","Data":"65adef64802bf3ca3d9cfb24cff08dea4f4031a9d48ef51c9baf9dd0b8607c03"} Jul 21 19:45:02.665996 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:02.665697 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" Jul 21 19:45:02.695841 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:02.695791 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" podStartSLOduration=1.849013384 podStartE2EDuration="3.69577525s" totalImagesPullingTime="1.846761866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:45:02.693301768 +0000 UTC m=+465.877743470" watchObservedRunningTime="2026-07-21 19:45:02.69577525 +0000 UTC m=+465.880216931" Jul 21 19:45:04.853221 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:04.853189 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r"] Jul 21 19:45:04.856770 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:04.856753 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:04.859801 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:04.859775 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vrj9n\"" Jul 21 19:45:04.879470 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:04.879441 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r"] Jul 21 19:45:04.936653 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:04.936621 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/223f0459-290e-4702-ba90-8091d23cbd90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" (UID: \"223f0459-290e-4702-ba90-8091d23cbd90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:04.936811 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:04.936716 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsbr\" (UniqueName: \"kubernetes.io/projected/223f0459-290e-4702-ba90-8091d23cbd90-kube-api-access-wwsbr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" (UID: \"223f0459-290e-4702-ba90-8091d23cbd90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:05.037392 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:05.037353 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/223f0459-290e-4702-ba90-8091d23cbd90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" (UID: \"223f0459-290e-4702-ba90-8091d23cbd90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:05.037583 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:05.037433 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsbr\" (UniqueName: \"kubernetes.io/projected/223f0459-290e-4702-ba90-8091d23cbd90-kube-api-access-wwsbr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" (UID: \"223f0459-290e-4702-ba90-8091d23cbd90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:05.037805 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:05.037780 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/223f0459-290e-4702-ba90-8091d23cbd90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" (UID: \"223f0459-290e-4702-ba90-8091d23cbd90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:05.058493 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:05.058457 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsbr\" (UniqueName: \"kubernetes.io/projected/223f0459-290e-4702-ba90-8091d23cbd90-kube-api-access-wwsbr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" (UID: \"223f0459-290e-4702-ba90-8091d23cbd90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:05.166984 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:05.166953 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:05.314650 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:05.314615 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r"] Jul 21 19:45:05.315419 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:45:05.315381 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223f0459_290e_4702_ba90_8091d23cbd90.slice/crio-3250fcfab6411284d949105ca92565c6d76445910f6b70186b2d0b7e0aa04934 WatchSource:0}: Error finding container 3250fcfab6411284d949105ca92565c6d76445910f6b70186b2d0b7e0aa04934: Status 404 returned error can't find the container with id 3250fcfab6411284d949105ca92565c6d76445910f6b70186b2d0b7e0aa04934 Jul 21 19:45:05.681199 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:05.681167 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" event={"ID":"223f0459-290e-4702-ba90-8091d23cbd90","Type":"ContainerStarted","Data":"3250fcfab6411284d949105ca92565c6d76445910f6b70186b2d0b7e0aa04934"} Jul 21 19:45:08.545078 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:08.545042 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg"] Jul 21 19:45:08.548720 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:08.548691 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" Jul 21 19:45:08.552194 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:08.552172 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fhbfx\"" Jul 21 19:45:08.559892 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:08.559853 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg"] Jul 21 19:45:08.670814 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:08.670766 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpsq\" (UniqueName: \"kubernetes.io/projected/28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed-kube-api-access-zgpsq\") pod \"authorino-operator-75cc8d5cb7-vd2sg\" (UID: \"28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" Jul 21 19:45:08.772276 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:08.772236 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpsq\" (UniqueName: \"kubernetes.io/projected/28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed-kube-api-access-zgpsq\") pod \"authorino-operator-75cc8d5cb7-vd2sg\" (UID: \"28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" Jul 21 19:45:08.781445 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:08.781409 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpsq\" (UniqueName: \"kubernetes.io/projected/28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed-kube-api-access-zgpsq\") pod \"authorino-operator-75cc8d5cb7-vd2sg\" (UID: \"28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" Jul 21 19:45:08.862164 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:08.862091 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" Jul 21 19:45:09.016284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:09.016253 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg"] Jul 21 19:45:09.017744 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:45:09.017717 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d3d2cd_aa2a_4e59_bcb0_e1f38ecb3aed.slice/crio-214115825c5ba160e01017c4dd7a6b933de9ffb238337034717e3186275db607 WatchSource:0}: Error finding container 214115825c5ba160e01017c4dd7a6b933de9ffb238337034717e3186275db607: Status 404 returned error can't find the container with id 214115825c5ba160e01017c4dd7a6b933de9ffb238337034717e3186275db607 Jul 21 19:45:09.711818 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:09.711774 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" event={"ID":"28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed","Type":"ContainerStarted","Data":"214115825c5ba160e01017c4dd7a6b933de9ffb238337034717e3186275db607"} Jul 21 19:45:11.725566 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:11.725523 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" event={"ID":"223f0459-290e-4702-ba90-8091d23cbd90","Type":"ContainerStarted","Data":"92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1"} Jul 21 19:45:11.726021 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:11.725658 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:11.732597 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:11.732572 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b"] Jul 21 19:45:11.739324 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:11.739302 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:11.743083 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:11.743062 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2dqhj\"" Jul 21 19:45:11.754113 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:11.754035 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b"] Jul 21 19:45:11.755638 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:11.755589 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" podStartSLOduration=2.410990424 podStartE2EDuration="7.75557128s" totalImagesPullingTime="5.344580856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:45:11.754638152 +0000 UTC m=+474.939079856" watchObservedRunningTime="2026-07-21 19:45:11.75557128 +0000 UTC m=+474.940012962" Jul 21 19:45:11.902111 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:11.902070 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lmn6\" (UniqueName: \"kubernetes.io/projected/e51b6d9b-7339-43bd-a22d-6b0fb5395253-kube-api-access-4lmn6\") pod \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" (UID: \"e51b6d9b-7339-43bd-a22d-6b0fb5395253\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:12.002739 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.002651 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lmn6\" (UniqueName: \"kubernetes.io/projected/e51b6d9b-7339-43bd-a22d-6b0fb5395253-kube-api-access-4lmn6\") pod \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" (UID: \"e51b6d9b-7339-43bd-a22d-6b0fb5395253\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:12.015690 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.015658 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lmn6\" (UniqueName: \"kubernetes.io/projected/e51b6d9b-7339-43bd-a22d-6b0fb5395253-kube-api-access-4lmn6\") pod \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" (UID: \"e51b6d9b-7339-43bd-a22d-6b0fb5395253\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:12.052930 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.052893 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:12.570702 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.570680 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b"] Jul 21 19:45:12.572976 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:45:12.572949 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51b6d9b_7339_43bd_a22d_6b0fb5395253.slice/crio-449ae94a56335bb2d5ee3fc95becfd96c673573f25a3d9a8f3f017d860cfdc40 WatchSource:0}: Error finding container 449ae94a56335bb2d5ee3fc95becfd96c673573f25a3d9a8f3f017d860cfdc40: Status 404 returned error can't find the container with id 449ae94a56335bb2d5ee3fc95becfd96c673573f25a3d9a8f3f017d860cfdc40 Jul 21 19:45:12.669904 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.669824 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-788fc66c5d-dmt65" podUID="3899a765-db3b-4ace-a0b2-b4024f1ea317" containerName="console" containerID="cri-o://f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac" gracePeriod=15 Jul 21 19:45:12.732044 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.731956 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" event={"ID":"28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed","Type":"ContainerStarted","Data":"299a3064f3dc77aa9c7804924eb0e954ff1ed7933965ce9521e4517773a45b62"} Jul 21 19:45:12.732444 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.732057 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" Jul 21 19:45:12.733369 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.733345 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" event={"ID":"e51b6d9b-7339-43bd-a22d-6b0fb5395253","Type":"ContainerStarted","Data":"449ae94a56335bb2d5ee3fc95becfd96c673573f25a3d9a8f3f017d860cfdc40"} Jul 21 19:45:12.752519 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.752462 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" podStartSLOduration=1.2855089880000001 podStartE2EDuration="4.752444729s" totalImagesPullingTime="3.466935741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:45:12.750695562 +0000 UTC m=+475.935137271" watchObservedRunningTime="2026-07-21 19:45:12.752444729 +0000 UTC m=+475.936886410" Jul 21 19:45:12.936183 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:12.936155 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:45:13.116198 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116159 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-oauth-serving-cert\") pod \"3899a765-db3b-4ace-a0b2-b4024f1ea317\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " Jul 21 19:45:13.116366 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116218 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-serving-cert\") pod \"3899a765-db3b-4ace-a0b2-b4024f1ea317\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " Jul 21 19:45:13.116366 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116290 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-oauth-config\") pod \"3899a765-db3b-4ace-a0b2-b4024f1ea317\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " Jul 21 19:45:13.116366 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116322 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-service-ca\") pod \"3899a765-db3b-4ace-a0b2-b4024f1ea317\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " Jul 21 19:45:13.116531 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116368 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfdqg\" (UniqueName: \"kubernetes.io/projected/3899a765-db3b-4ace-a0b2-b4024f1ea317-kube-api-access-qfdqg\") pod \"3899a765-db3b-4ace-a0b2-b4024f1ea317\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " Jul 21 19:45:13.116531 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116406 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-config\") pod \"3899a765-db3b-4ace-a0b2-b4024f1ea317\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " Jul 21 19:45:13.116531 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116498 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-trusted-ca-bundle\") pod \"3899a765-db3b-4ace-a0b2-b4024f1ea317\" (UID: \"3899a765-db3b-4ace-a0b2-b4024f1ea317\") " Jul 21 19:45:13.116686 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116585 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3899a765-db3b-4ace-a0b2-b4024f1ea317" (UID: "3899a765-db3b-4ace-a0b2-b4024f1ea317"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:45:13.116686 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116660 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-service-ca" (OuterVolumeSpecName: "service-ca") pod "3899a765-db3b-4ace-a0b2-b4024f1ea317" (UID: "3899a765-db3b-4ace-a0b2-b4024f1ea317"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:45:13.116935 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116854 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-config" (OuterVolumeSpecName: "console-config") pod "3899a765-db3b-4ace-a0b2-b4024f1ea317" (UID: "3899a765-db3b-4ace-a0b2-b4024f1ea317"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:45:13.117044 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116940 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-oauth-serving-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:13.117044 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.116962 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-service-ca\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:13.117150 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.117120 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3899a765-db3b-4ace-a0b2-b4024f1ea317" (UID: "3899a765-db3b-4ace-a0b2-b4024f1ea317"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 19:45:13.118467 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.118435 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3899a765-db3b-4ace-a0b2-b4024f1ea317" (UID: "3899a765-db3b-4ace-a0b2-b4024f1ea317"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:45:13.118615 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.118498 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3899a765-db3b-4ace-a0b2-b4024f1ea317" (UID: "3899a765-db3b-4ace-a0b2-b4024f1ea317"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 19:45:13.118615 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.118503 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3899a765-db3b-4ace-a0b2-b4024f1ea317-kube-api-access-qfdqg" (OuterVolumeSpecName: "kube-api-access-qfdqg") pod "3899a765-db3b-4ace-a0b2-b4024f1ea317" (UID: "3899a765-db3b-4ace-a0b2-b4024f1ea317"). InnerVolumeSpecName "kube-api-access-qfdqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:45:13.217723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.217688 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-serving-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:13.217723 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.217725 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-oauth-config\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:13.217966 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.217741 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdqg\" (UniqueName: \"kubernetes.io/projected/3899a765-db3b-4ace-a0b2-b4024f1ea317-kube-api-access-qfdqg\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:13.217966 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.217757 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-console-config\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:13.217966 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.217770 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3899a765-db3b-4ace-a0b2-b4024f1ea317-trusted-ca-bundle\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:13.673117 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.673075 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bx9bm" Jul 21 19:45:13.741998 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.741945 2557 generic.go:356] "Generic (PLEG): container finished" podID="3899a765-db3b-4ace-a0b2-b4024f1ea317" containerID="f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac" exitCode=0 Jul 21 19:45:13.742498 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.742022 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788fc66c5d-dmt65" Jul 21 19:45:13.742498 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.742082 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788fc66c5d-dmt65" event={"ID":"3899a765-db3b-4ace-a0b2-b4024f1ea317","Type":"ContainerDied","Data":"f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac"} Jul 21 19:45:13.742498 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.742129 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788fc66c5d-dmt65" event={"ID":"3899a765-db3b-4ace-a0b2-b4024f1ea317","Type":"ContainerDied","Data":"43c232bae16b3e37a13f19a4d3355c289da719974a483273be31b2fb682cf753"} Jul 21 19:45:13.742498 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.742155 2557 scope.go:117] "RemoveContainer" containerID="f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac" Jul 21 19:45:13.758715 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.758617 2557 scope.go:117] "RemoveContainer" containerID="f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac" Jul 21 19:45:13.759457 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:13.759428 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac\": container with ID starting with f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac not found: ID does not exist" containerID="f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac" Jul 21 19:45:13.759570 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.759468 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac"} err="failed to get container status \"f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac\": rpc error: code = NotFound desc = could not find container \"f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac\": container with ID starting with f7afeaf63617bb2b92f465e16d3e3923a8690362d4aa9831a0bcc63d293d9dac not found: ID does not exist" Jul 21 19:45:13.778264 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.778236 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-788fc66c5d-dmt65"] Jul 21 19:45:13.786602 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:13.786576 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-788fc66c5d-dmt65"] Jul 21 19:45:14.505785 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:14.505700 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3899a765-db3b-4ace-a0b2-b4024f1ea317" path="/var/lib/kubelet/pods/3899a765-db3b-4ace-a0b2-b4024f1ea317/volumes" Jul 21 19:45:14.748506 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:14.748469 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" event={"ID":"e51b6d9b-7339-43bd-a22d-6b0fb5395253","Type":"ContainerStarted","Data":"bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be"} Jul 21 19:45:14.748954 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:14.748531 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:14.783280 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:14.783183 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" podStartSLOduration=2.199778349 podStartE2EDuration="3.783166175s" totalImagesPullingTime="1.583387826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:45:14.780837608 +0000 UTC m=+477.965279321" watchObservedRunningTime="2026-07-21 19:45:14.783166175 +0000 UTC m=+477.967608123" Jul 21 19:45:22.736456 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:22.736426 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:23.745067 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:23.745038 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vd2sg" Jul 21 19:45:24.494637 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.494598 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r"] Jul 21 19:45:24.494831 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.494807 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" podUID="223f0459-290e-4702-ba90-8091d23cbd90" containerName="manager" containerID="cri-o://92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1" gracePeriod=2 Jul 21 19:45:24.508838 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.508809 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r"] Jul 21 19:45:24.514001 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.513968 2557 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"223f0459-290e-4702-ba90-8091d23cbd90\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-zc68r\": pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.534814 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.534789 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77"] Jul 21 19:45:24.535168 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.535154 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223f0459-290e-4702-ba90-8091d23cbd90" containerName="manager" Jul 21 19:45:24.535168 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.535169 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="223f0459-290e-4702-ba90-8091d23cbd90" containerName="manager" Jul 21 19:45:24.535254 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.535187 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3899a765-db3b-4ace-a0b2-b4024f1ea317" containerName="console" Jul 21 19:45:24.535254 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.535192 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3899a765-db3b-4ace-a0b2-b4024f1ea317" containerName="console" Jul 21 19:45:24.535322 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.535255 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="223f0459-290e-4702-ba90-8091d23cbd90" containerName="manager" Jul 21 19:45:24.535322 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.535267 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3899a765-db3b-4ace-a0b2-b4024f1ea317" containerName="console" Jul 21 19:45:24.538397 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.538381 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76"] Jul 21 19:45:24.538517 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.538499 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:24.541665 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.541646 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b"] Jul 21 19:45:24.541825 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.541804 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" containerName="manager" containerID="cri-o://bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be" gracePeriod=2 Jul 21 19:45:24.541904 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.541812 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:24.543672 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.543645 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.555053 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.555000 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b"] Jul 21 19:45:24.557721 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.557698 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77"] Jul 21 19:45:24.560597 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.560576 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76"] Jul 21 19:45:24.567342 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.567326 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4"] Jul 21 19:45:24.567800 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.567785 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" containerName="manager" Jul 21 19:45:24.567852 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.567802 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" containerName="manager" Jul 21 19:45:24.567906 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.567893 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" containerName="manager" Jul 21 19:45:24.571137 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.571117 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" Jul 21 19:45:24.572911 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.572848 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.576388 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.576363 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.585666 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.585643 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4"] Jul 21 19:45:24.608692 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.608662 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8018e0b4-fc27-4eb0-80ea-2d516237118d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wjs76\" (UID: \"8018e0b4-fc27-4eb0-80ea-2d516237118d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:24.608822 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.608708 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnq89\" (UniqueName: \"kubernetes.io/projected/7cce3fe9-bf01-4740-9049-eaa146de312b-kube-api-access-tnq89\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9p77\" (UID: \"7cce3fe9-bf01-4740-9049-eaa146de312b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:24.608822 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.608760 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wphnc\" (UniqueName: \"kubernetes.io/projected/8018e0b4-fc27-4eb0-80ea-2d516237118d-kube-api-access-wphnc\") pod \"kuadrant-operator-controller-manager-86878c769f-wjs76\" (UID: \"8018e0b4-fc27-4eb0-80ea-2d516237118d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:24.608935 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.608852 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vxvg\" (UniqueName: \"kubernetes.io/projected/66329117-8064-4222-9ef2-2b96f79ebed0-kube-api-access-5vxvg\") pod \"limitador-operator-controller-manager-859d7b78bd-5ssp4\" (UID: \"66329117-8064-4222-9ef2-2b96f79ebed0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" Jul 21 19:45:24.608997 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.608975 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cce3fe9-bf01-4740-9049-eaa146de312b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9p77\" (UID: \"7cce3fe9-bf01-4740-9049-eaa146de312b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:24.625790 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.625760 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.709991 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.709946 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vxvg\" (UniqueName: \"kubernetes.io/projected/66329117-8064-4222-9ef2-2b96f79ebed0-kube-api-access-5vxvg\") pod \"limitador-operator-controller-manager-859d7b78bd-5ssp4\" (UID: \"66329117-8064-4222-9ef2-2b96f79ebed0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" Jul 21 19:45:24.710148 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.710021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cce3fe9-bf01-4740-9049-eaa146de312b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9p77\" (UID: \"7cce3fe9-bf01-4740-9049-eaa146de312b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:24.710148 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.710065 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8018e0b4-fc27-4eb0-80ea-2d516237118d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wjs76\" (UID: \"8018e0b4-fc27-4eb0-80ea-2d516237118d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:24.710148 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.710096 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnq89\" (UniqueName: \"kubernetes.io/projected/7cce3fe9-bf01-4740-9049-eaa146de312b-kube-api-access-tnq89\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9p77\" (UID: \"7cce3fe9-bf01-4740-9049-eaa146de312b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:24.710148 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.710126 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wphnc\" (UniqueName: \"kubernetes.io/projected/8018e0b4-fc27-4eb0-80ea-2d516237118d-kube-api-access-wphnc\") pod \"kuadrant-operator-controller-manager-86878c769f-wjs76\" (UID: \"8018e0b4-fc27-4eb0-80ea-2d516237118d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:24.710413 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.710373 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cce3fe9-bf01-4740-9049-eaa146de312b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9p77\" (UID: \"7cce3fe9-bf01-4740-9049-eaa146de312b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:24.710737 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.710718 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8018e0b4-fc27-4eb0-80ea-2d516237118d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wjs76\" (UID: \"8018e0b4-fc27-4eb0-80ea-2d516237118d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:24.731817 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.731754 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vxvg\" (UniqueName: \"kubernetes.io/projected/66329117-8064-4222-9ef2-2b96f79ebed0-kube-api-access-5vxvg\") pod \"limitador-operator-controller-manager-859d7b78bd-5ssp4\" (UID: \"66329117-8064-4222-9ef2-2b96f79ebed0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" Jul 21 19:45:24.733835 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.733690 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphnc\" (UniqueName: \"kubernetes.io/projected/8018e0b4-fc27-4eb0-80ea-2d516237118d-kube-api-access-wphnc\") pod \"kuadrant-operator-controller-manager-86878c769f-wjs76\" (UID: \"8018e0b4-fc27-4eb0-80ea-2d516237118d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:24.734209 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.734148 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnq89\" (UniqueName: \"kubernetes.io/projected/7cce3fe9-bf01-4740-9049-eaa146de312b-kube-api-access-tnq89\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9p77\" (UID: \"7cce3fe9-bf01-4740-9049-eaa146de312b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:24.786495 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.786467 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.788610 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.788592 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.788927 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.788906 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.791756 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.791736 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.794293 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.794275 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.797493 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.797475 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.811067 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.811043 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwsbr\" (UniqueName: \"kubernetes.io/projected/223f0459-290e-4702-ba90-8091d23cbd90-kube-api-access-wwsbr\") pod \"223f0459-290e-4702-ba90-8091d23cbd90\" (UID: \"223f0459-290e-4702-ba90-8091d23cbd90\") " Jul 21 19:45:24.811204 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.811085 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/223f0459-290e-4702-ba90-8091d23cbd90-extensions-socket-volume\") pod \"223f0459-290e-4702-ba90-8091d23cbd90\" (UID: \"223f0459-290e-4702-ba90-8091d23cbd90\") " Jul 21 19:45:24.811204 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.811104 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lmn6\" (UniqueName: \"kubernetes.io/projected/e51b6d9b-7339-43bd-a22d-6b0fb5395253-kube-api-access-4lmn6\") pod \"e51b6d9b-7339-43bd-a22d-6b0fb5395253\" (UID: \"e51b6d9b-7339-43bd-a22d-6b0fb5395253\") " Jul 21 19:45:24.811459 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.811429 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223f0459-290e-4702-ba90-8091d23cbd90-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "223f0459-290e-4702-ba90-8091d23cbd90" (UID: "223f0459-290e-4702-ba90-8091d23cbd90"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:45:24.813373 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.813350 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51b6d9b-7339-43bd-a22d-6b0fb5395253-kube-api-access-4lmn6" (OuterVolumeSpecName: "kube-api-access-4lmn6") pod "e51b6d9b-7339-43bd-a22d-6b0fb5395253" (UID: "e51b6d9b-7339-43bd-a22d-6b0fb5395253"). InnerVolumeSpecName "kube-api-access-4lmn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:45:24.813373 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.813362 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223f0459-290e-4702-ba90-8091d23cbd90-kube-api-access-wwsbr" (OuterVolumeSpecName: "kube-api-access-wwsbr") pod "223f0459-290e-4702-ba90-8091d23cbd90" (UID: "223f0459-290e-4702-ba90-8091d23cbd90"). InnerVolumeSpecName "kube-api-access-wwsbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:45:24.813602 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.813584 2557 generic.go:356] "Generic (PLEG): container finished" podID="223f0459-290e-4702-ba90-8091d23cbd90" containerID="92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1" exitCode=0 Jul 21 19:45:24.813666 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.813640 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.813716 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.813680 2557 scope.go:117] "RemoveContainer" containerID="92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1" Jul 21 19:45:24.815106 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.815085 2557 generic.go:356] "Generic (PLEG): container finished" podID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" containerID="bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be" exitCode=0 Jul 21 19:45:24.815222 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.815122 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.816538 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.816510 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.820028 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.820000 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.821993 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.821970 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.823893 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.823854 2557 scope.go:117] "RemoveContainer" containerID="92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1" Jul 21 19:45:24.824092 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.824057 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.824188 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.824154 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1\": container with ID starting with 92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1 not found: ID does not exist" containerID="92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1" Jul 21 19:45:24.824252 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.824180 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1"} err="failed to get container status \"92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1\": rpc error: code = NotFound desc = could not find container \"92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1\": container with ID starting with 92d5bea19f913f9742ef73f85b0a9a0653582781a13fbe50c1646d793a5a27b1 not found: ID does not exist" Jul 21 19:45:24.824252 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.824205 2557 scope.go:117] "RemoveContainer" containerID="bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be" Jul 21 19:45:24.826782 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.826762 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.829397 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.829368 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.832780 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.832746 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:24.834580 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.834561 2557 scope.go:117] "RemoveContainer" containerID="bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be" Jul 21 19:45:24.834826 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.834805 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:24.834910 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:24.834892 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be\": container with ID starting with bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be not found: ID does not exist" containerID="bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be" Jul 21 19:45:24.834958 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.834917 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be"} err="failed to get container status \"bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be\": rpc error: code = NotFound desc = could not find container \"bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be\": container with ID starting with bfa6e6eab5005aae837698f1a97d08af997a1f17c94ed873f7d00697189386be not found: ID does not exist" Jul 21 19:45:24.912764 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.912736 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwsbr\" (UniqueName: \"kubernetes.io/projected/223f0459-290e-4702-ba90-8091d23cbd90-kube-api-access-wwsbr\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:24.912764 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.912762 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/223f0459-290e-4702-ba90-8091d23cbd90-extensions-socket-volume\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:24.912971 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.912773 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lmn6\" (UniqueName: \"kubernetes.io/projected/e51b6d9b-7339-43bd-a22d-6b0fb5395253-kube-api-access-4lmn6\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:24.927656 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.927628 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:24.934207 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.934180 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:24.940225 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:24.940202 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" Jul 21 19:45:25.097344 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.097175 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77"] Jul 21 19:45:25.100494 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:45:25.100461 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cce3fe9_bf01_4740_9049_eaa146de312b.slice/crio-af0dd6532dae61c25d42237dbaec60f3068670531a29174fab2287a5067d4100 WatchSource:0}: Error finding container af0dd6532dae61c25d42237dbaec60f3068670531a29174fab2287a5067d4100: Status 404 returned error can't find the container with id af0dd6532dae61c25d42237dbaec60f3068670531a29174fab2287a5067d4100 Jul 21 19:45:25.112585 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.112561 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76"] Jul 21 19:45:25.116180 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:45:25.116138 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8018e0b4_fc27_4eb0_80ea_2d516237118d.slice/crio-52f7d4294fd344410fda8b3e83b100f8abdcce0f5d3a1e763f2402c598595a7b WatchSource:0}: Error finding container 52f7d4294fd344410fda8b3e83b100f8abdcce0f5d3a1e763f2402c598595a7b: Status 404 returned error can't find the container with id 52f7d4294fd344410fda8b3e83b100f8abdcce0f5d3a1e763f2402c598595a7b Jul 21 19:45:25.132217 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.132193 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4"] Jul 21 19:45:25.134120 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:45:25.134091 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66329117_8064_4222_9ef2_2b96f79ebed0.slice/crio-c9c39eab704a10e1d7f1a34b1cb43df9cc59232ebfbaaad08767095087a348de WatchSource:0}: Error finding container c9c39eab704a10e1d7f1a34b1cb43df9cc59232ebfbaaad08767095087a348de: Status 404 returned error can't find the container with id c9c39eab704a10e1d7f1a34b1cb43df9cc59232ebfbaaad08767095087a348de Jul 21 19:45:25.822479 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.822375 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" event={"ID":"66329117-8064-4222-9ef2-2b96f79ebed0","Type":"ContainerStarted","Data":"1af85974e07eea8fa1f3223b9b99d991f883fd1a27745c6369ee48fc9865dab0"} Jul 21 19:45:25.822479 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.822452 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" event={"ID":"66329117-8064-4222-9ef2-2b96f79ebed0","Type":"ContainerStarted","Data":"c9c39eab704a10e1d7f1a34b1cb43df9cc59232ebfbaaad08767095087a348de"} Jul 21 19:45:25.822951 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.822514 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" Jul 21 19:45:25.824080 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.824055 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" event={"ID":"8018e0b4-fc27-4eb0-80ea-2d516237118d","Type":"ContainerStarted","Data":"2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99"} Jul 21 19:45:25.824213 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.824100 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:25.824213 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.824124 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" event={"ID":"8018e0b4-fc27-4eb0-80ea-2d516237118d","Type":"ContainerStarted","Data":"52f7d4294fd344410fda8b3e83b100f8abdcce0f5d3a1e763f2402c598595a7b"} Jul 21 19:45:25.826381 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.826363 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" event={"ID":"7cce3fe9-bf01-4740-9049-eaa146de312b","Type":"ContainerStarted","Data":"767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8"} Jul 21 19:45:25.826488 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.826400 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" event={"ID":"7cce3fe9-bf01-4740-9049-eaa146de312b","Type":"ContainerStarted","Data":"af0dd6532dae61c25d42237dbaec60f3068670531a29174fab2287a5067d4100"} Jul 21 19:45:25.826488 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.826430 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:25.865087 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.865030 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" podStartSLOduration=1.865013313 podStartE2EDuration="1.865013313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:45:25.862903782 +0000 UTC m=+489.047345482" watchObservedRunningTime="2026-07-21 19:45:25.865013313 +0000 UTC m=+489.049454994" Jul 21 19:45:25.865575 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:25.865550 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:25.868052 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:25.868025 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:25.870548 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:25.870525 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5wj8b\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wj8b" Jul 21 19:45:25.913905 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.913841 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" podStartSLOduration=1.913824024 podStartE2EDuration="1.913824024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:45:25.91219493 +0000 UTC m=+489.096636625" watchObservedRunningTime="2026-07-21 19:45:25.913824024 +0000 UTC m=+489.098265705" Jul 21 19:45:25.914929 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:25.914905 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zc68r\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="223f0459-290e-4702-ba90-8091d23cbd90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zc68r" Jul 21 19:45:25.953342 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:25.953286 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" podStartSLOduration=1.95326947 podStartE2EDuration="1.95326947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:45:25.951698762 +0000 UTC m=+489.136140474" watchObservedRunningTime="2026-07-21 19:45:25.95326947 +0000 UTC m=+489.137711150" Jul 21 19:45:26.512601 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:26.512556 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223f0459-290e-4702-ba90-8091d23cbd90" path="/var/lib/kubelet/pods/223f0459-290e-4702-ba90-8091d23cbd90/volumes" Jul 21 19:45:26.513065 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:26.513046 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e51b6d9b-7339-43bd-a22d-6b0fb5395253" path="/var/lib/kubelet/pods/e51b6d9b-7339-43bd-a22d-6b0fb5395253/volumes" Jul 21 19:45:36.834016 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:36.833969 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5ssp4" Jul 21 19:45:36.834512 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:36.834129 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:36.834512 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:36.834165 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:45:37.032395 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.032362 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77"] Jul 21 19:45:37.032564 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.032541 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" podUID="7cce3fe9-bf01-4740-9049-eaa146de312b" containerName="manager" containerID="cri-o://767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8" gracePeriod=10 Jul 21 19:45:37.323781 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.323756 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:37.425536 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.425506 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cce3fe9-bf01-4740-9049-eaa146de312b-extensions-socket-volume\") pod \"7cce3fe9-bf01-4740-9049-eaa146de312b\" (UID: \"7cce3fe9-bf01-4740-9049-eaa146de312b\") " Jul 21 19:45:37.425696 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.425598 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnq89\" (UniqueName: \"kubernetes.io/projected/7cce3fe9-bf01-4740-9049-eaa146de312b-kube-api-access-tnq89\") pod \"7cce3fe9-bf01-4740-9049-eaa146de312b\" (UID: \"7cce3fe9-bf01-4740-9049-eaa146de312b\") " Jul 21 19:45:37.425955 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.425935 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce3fe9-bf01-4740-9049-eaa146de312b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7cce3fe9-bf01-4740-9049-eaa146de312b" (UID: "7cce3fe9-bf01-4740-9049-eaa146de312b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:45:37.427576 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.427554 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cce3fe9-bf01-4740-9049-eaa146de312b-kube-api-access-tnq89" (OuterVolumeSpecName: "kube-api-access-tnq89") pod "7cce3fe9-bf01-4740-9049-eaa146de312b" (UID: "7cce3fe9-bf01-4740-9049-eaa146de312b"). InnerVolumeSpecName "kube-api-access-tnq89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:45:37.526667 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.526634 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cce3fe9-bf01-4740-9049-eaa146de312b-extensions-socket-volume\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:37.526667 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.526662 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnq89\" (UniqueName: \"kubernetes.io/projected/7cce3fe9-bf01-4740-9049-eaa146de312b-kube-api-access-tnq89\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:45:37.890506 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.890474 2557 generic.go:356] "Generic (PLEG): container finished" podID="7cce3fe9-bf01-4740-9049-eaa146de312b" containerID="767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8" exitCode=0 Jul 21 19:45:37.890916 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.890515 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" Jul 21 19:45:37.890916 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.890557 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" event={"ID":"7cce3fe9-bf01-4740-9049-eaa146de312b","Type":"ContainerDied","Data":"767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8"} Jul 21 19:45:37.890916 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.890609 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77" event={"ID":"7cce3fe9-bf01-4740-9049-eaa146de312b","Type":"ContainerDied","Data":"af0dd6532dae61c25d42237dbaec60f3068670531a29174fab2287a5067d4100"} Jul 21 19:45:37.890916 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.890638 2557 scope.go:117] "RemoveContainer" containerID="767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8" Jul 21 19:45:37.899640 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.899624 2557 scope.go:117] "RemoveContainer" containerID="767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8" Jul 21 19:45:37.899949 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:45:37.899932 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8\": container with ID starting with 767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8 not found: ID does not exist" containerID="767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8" Jul 21 19:45:37.900014 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.899957 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8"} err="failed to get container status \"767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8\": rpc error: code = NotFound desc = could not find container \"767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8\": container with ID starting with 767d2cd22ba5cd7168728bac745f5e559d42c221fa241eda30709f0f0f32dcd8 not found: ID does not exist" Jul 21 19:45:37.916182 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.916156 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77"] Jul 21 19:45:37.924634 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:37.924613 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9p77"] Jul 21 19:45:38.505285 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:38.505254 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cce3fe9-bf01-4740-9049-eaa146de312b" path="/var/lib/kubelet/pods/7cce3fe9-bf01-4740-9049-eaa146de312b/volumes" Jul 21 19:45:53.473284 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.473242 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d"] Jul 21 19:45:53.473699 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.473645 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cce3fe9-bf01-4740-9049-eaa146de312b" containerName="manager" Jul 21 19:45:53.473699 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.473656 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cce3fe9-bf01-4740-9049-eaa146de312b" containerName="manager" Jul 21 19:45:53.473790 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.473716 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cce3fe9-bf01-4740-9049-eaa146de312b" containerName="manager" Jul 21 19:45:53.476814 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.476799 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.481683 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.481657 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2h4l8\"" Jul 21 19:45:53.496379 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.496349 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d"] Jul 21 19:45:53.570241 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.570203 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.570412 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.570249 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.570412 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.570269 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e417bcb3-4037-4ede-93ed-a712a90a8c70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.570412 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.570343 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9szc8\" (UniqueName: \"kubernetes.io/projected/e417bcb3-4037-4ede-93ed-a712a90a8c70-kube-api-access-9szc8\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.570412 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.570367 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.570412 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.570386 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.570582 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.570431 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.570582 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.570465 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.570582 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.570492 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.671377 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671329 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.671377 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671369 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e417bcb3-4037-4ede-93ed-a712a90a8c70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671406 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szc8\" (UniqueName: \"kubernetes.io/projected/e417bcb3-4037-4ede-93ed-a712a90a8c70-kube-api-access-9szc8\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671432 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671456 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671489 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671536 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671568 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671772 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.671846 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.672040 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.672141 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e417bcb3-4037-4ede-93ed-a712a90a8c70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.672328 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.672212 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.673725 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.673709 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.674447 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.674406 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.680721 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.680677 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szc8\" (UniqueName: \"kubernetes.io/projected/e417bcb3-4037-4ede-93ed-a712a90a8c70-kube-api-access-9szc8\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.681024 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.681007 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e417bcb3-4037-4ede-93ed-a712a90a8c70-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-m7s7d\" (UID: \"e417bcb3-4037-4ede-93ed-a712a90a8c70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.786988 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.786907 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:53.933078 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.933052 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d"] Jul 21 19:45:53.935166 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:45:53.935142 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode417bcb3_4037_4ede_93ed_a712a90a8c70.slice/crio-61d49b9ac162f9258ff60e50ecfd7d05ebc112c2743ce75ea719f8b4e94e0741 WatchSource:0}: Error finding container 61d49b9ac162f9258ff60e50ecfd7d05ebc112c2743ce75ea719f8b4e94e0741: Status 404 returned error can't find the container with id 61d49b9ac162f9258ff60e50ecfd7d05ebc112c2743ce75ea719f8b4e94e0741 Jul 21 19:45:53.938179 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.938146 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:45:53.938258 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.938226 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:45:53.938299 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.938265 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:45:53.975697 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:53.975668 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" event={"ID":"e417bcb3-4037-4ede-93ed-a712a90a8c70","Type":"ContainerStarted","Data":"61d49b9ac162f9258ff60e50ecfd7d05ebc112c2743ce75ea719f8b4e94e0741"} Jul 21 19:45:54.982782 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:54.982746 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" event={"ID":"e417bcb3-4037-4ede-93ed-a712a90a8c70","Type":"ContainerStarted","Data":"e9a5f2f334a03a58c83f87951742e8757a1768c12e20f389a7784f22e15c6e21"} Jul 21 19:45:55.008557 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:55.008495 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" podStartSLOduration=2.008478441 podStartE2EDuration="2.008478441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:45:55.005166374 +0000 UTC m=+518.189608076" watchObservedRunningTime="2026-07-21 19:45:55.008478441 +0000 UTC m=+518.192920209" Jul 21 19:45:55.787553 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:55.787514 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:55.792653 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:55.792626 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:55.988464 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:55.988429 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:55.989386 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:55.989363 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:45:56.506585 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.506556 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:45:56.509998 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.509982 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:45:56.512961 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.512939 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zcmqw\"" Jul 21 19:45:56.513219 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.513200 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 19:45:56.521492 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.521468 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:45:56.554691 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.554663 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:45:56.598199 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.598172 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/59728d01-a27c-4881-ad06-3c0eaaa8df75-config-file\") pod \"limitador-limitador-54d6d988d5-n7rdc\" (UID: \"59728d01-a27c-4881-ad06-3c0eaaa8df75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:45:56.598352 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.598296 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82tbl\" (UniqueName: \"kubernetes.io/projected/59728d01-a27c-4881-ad06-3c0eaaa8df75-kube-api-access-82tbl\") pod \"limitador-limitador-54d6d988d5-n7rdc\" (UID: \"59728d01-a27c-4881-ad06-3c0eaaa8df75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:45:56.699197 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.699160 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82tbl\" (UniqueName: \"kubernetes.io/projected/59728d01-a27c-4881-ad06-3c0eaaa8df75-kube-api-access-82tbl\") pod \"limitador-limitador-54d6d988d5-n7rdc\" (UID: \"59728d01-a27c-4881-ad06-3c0eaaa8df75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:45:56.699353 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.699232 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/59728d01-a27c-4881-ad06-3c0eaaa8df75-config-file\") pod \"limitador-limitador-54d6d988d5-n7rdc\" (UID: \"59728d01-a27c-4881-ad06-3c0eaaa8df75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:45:56.699808 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.699790 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/59728d01-a27c-4881-ad06-3c0eaaa8df75-config-file\") pod \"limitador-limitador-54d6d988d5-n7rdc\" (UID: \"59728d01-a27c-4881-ad06-3c0eaaa8df75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:45:56.708420 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.708393 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82tbl\" (UniqueName: \"kubernetes.io/projected/59728d01-a27c-4881-ad06-3c0eaaa8df75-kube-api-access-82tbl\") pod \"limitador-limitador-54d6d988d5-n7rdc\" (UID: \"59728d01-a27c-4881-ad06-3c0eaaa8df75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:45:56.822039 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.821961 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:45:56.990430 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:45:56.990401 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59728d01_a27c_4881_ad06_3c0eaaa8df75.slice/crio-40d6b492b35e0dd26bb84afbce477a2c0d8832c6133f5b44352c969c23a603f8 WatchSource:0}: Error finding container 40d6b492b35e0dd26bb84afbce477a2c0d8832c6133f5b44352c969c23a603f8: Status 404 returned error can't find the container with id 40d6b492b35e0dd26bb84afbce477a2c0d8832c6133f5b44352c969c23a603f8 Jul 21 19:45:56.990928 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:56.990888 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:45:58.003222 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:45:58.003178 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" event={"ID":"59728d01-a27c-4881-ad06-3c0eaaa8df75","Type":"ContainerStarted","Data":"40d6b492b35e0dd26bb84afbce477a2c0d8832c6133f5b44352c969c23a603f8"} Jul 21 19:46:00.014795 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:46:00.014753 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" event={"ID":"59728d01-a27c-4881-ad06-3c0eaaa8df75","Type":"ContainerStarted","Data":"68018ec89997fdfa565b7f1ece0332888d57b5066616e467ed3727f8adc3838b"} Jul 21 19:46:00.015221 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:46:00.014869 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:46:00.036455 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:46:00.036408 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" podStartSLOduration=1.514870688 podStartE2EDuration="4.036393498s" totalImagesPullingTime="2.52152281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:46:00.034956698 +0000 UTC m=+523.219398400" watchObservedRunningTime="2026-07-21 19:46:00.036393498 +0000 UTC m=+523.220835202" Jul 21 19:46:11.020708 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:46:11.020621 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-n7rdc" Jul 21 19:47:09.787259 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:09.787223 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:47:17.154616 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:17.154584 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:47:17.159711 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:17.159688 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:47:17.420906 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:17.420814 2557 generic.go:356] "Generic (PLEG): container finished" podID="e417bcb3-4037-4ede-93ed-a712a90a8c70" containerID="e9a5f2f334a03a58c83f87951742e8757a1768c12e20f389a7784f22e15c6e21" exitCode=0 Jul 21 19:47:17.421083 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:17.420903 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" event={"ID":"e417bcb3-4037-4ede-93ed-a712a90a8c70","Type":"ContainerDied","Data":"e9a5f2f334a03a58c83f87951742e8757a1768c12e20f389a7784f22e15c6e21"} Jul 21 19:47:17.421307 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:17.421293 2557 scope.go:117] "RemoveContainer" containerID="e9a5f2f334a03a58c83f87951742e8757a1768c12e20f389a7784f22e15c6e21" Jul 21 19:47:17.422282 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:17.422252 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:47:17.422395 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:17.422315 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:47:17.422395 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:17.422344 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 19:47:18.427688 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:18.427652 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" event={"ID":"e417bcb3-4037-4ede-93ed-a712a90a8c70","Type":"ContainerStarted","Data":"19ad2bb6eecf8b46fa1a6dcfc7b893ef638f89d34c78f3e8afbc44bce27c77dd"} Jul 21 19:47:18.787414 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:18.787324 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:47:18.787900 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:18.787588 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-m7s7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 21 19:47:18.787900 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:18.787646 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" podUID="e417bcb3-4037-4ede-93ed-a712a90a8c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 21 19:47:19.787759 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:19.787723 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-m7s7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 21 19:47:19.788166 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:19.787803 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" podUID="e417bcb3-4037-4ede-93ed-a712a90a8c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 21 19:47:20.788155 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:20.788118 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-m7s7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 21 19:47:20.788534 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:20.788183 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" podUID="e417bcb3-4037-4ede-93ed-a712a90a8c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 21 19:47:21.788070 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:21.788029 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-m7s7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 21 19:47:21.788231 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:21.788100 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" podUID="e417bcb3-4037-4ede-93ed-a712a90a8c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 21 19:47:22.792007 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:22.791978 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:47:22.792454 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:22.792282 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:47:22.793090 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:22.793069 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-m7s7d" Jul 21 19:47:26.199706 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:26.199670 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:47:34.405391 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:34.405347 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:47:47.595082 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:47.594987 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:47:56.802978 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:47:56.802944 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:48:04.703905 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.703857 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c"] Jul 21 19:48:04.707521 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.707504 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.710991 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.710960 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 19:48:04.711105 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.710991 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 19:48:04.711267 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.711245 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wn54b\"" Jul 21 19:48:04.712169 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.712153 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 21 19:48:04.717693 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.717669 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c"] Jul 21 19:48:04.769430 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.769397 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqnsb\" (UniqueName: \"kubernetes.io/projected/16fbb2f2-02fc-461d-84bd-317600ee76c5-kube-api-access-jqnsb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.769604 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.769461 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.769604 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.769516 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16fbb2f2-02fc-461d-84bd-317600ee76c5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.769604 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.769557 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.769604 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.769580 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.769801 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.769651 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.871166 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.871130 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.871356 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.871211 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqnsb\" (UniqueName: \"kubernetes.io/projected/16fbb2f2-02fc-461d-84bd-317600ee76c5-kube-api-access-jqnsb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.871356 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.871246 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.871356 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.871264 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16fbb2f2-02fc-461d-84bd-317600ee76c5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.871356 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.871292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.871356 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.871321 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.871691 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.871582 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.871691 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.871622 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.871804 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.871693 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.873654 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.873629 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16fbb2f2-02fc-461d-84bd-317600ee76c5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.874038 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.874022 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16fbb2f2-02fc-461d-84bd-317600ee76c5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:04.879611 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:04.879590 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqnsb\" (UniqueName: \"kubernetes.io/projected/16fbb2f2-02fc-461d-84bd-317600ee76c5-kube-api-access-jqnsb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v5r5c\" (UID: \"16fbb2f2-02fc-461d-84bd-317600ee76c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:05.019098 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:05.019007 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:05.157814 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:05.157779 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c"] Jul 21 19:48:05.160976 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:48:05.160938 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16fbb2f2_02fc_461d_84bd_317600ee76c5.slice/crio-0045479cac096962c2b79c29934a23d6c212069aafb80065650b7c2fea0875eb WatchSource:0}: Error finding container 0045479cac096962c2b79c29934a23d6c212069aafb80065650b7c2fea0875eb: Status 404 returned error can't find the container with id 0045479cac096962c2b79c29934a23d6c212069aafb80065650b7c2fea0875eb Jul 21 19:48:05.162201 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:05.162184 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 19:48:05.686783 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:05.686737 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" event={"ID":"16fbb2f2-02fc-461d-84bd-317600ee76c5","Type":"ContainerStarted","Data":"0045479cac096962c2b79c29934a23d6c212069aafb80065650b7c2fea0875eb"} Jul 21 19:48:06.694325 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:06.694289 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" event={"ID":"16fbb2f2-02fc-461d-84bd-317600ee76c5","Type":"ContainerStarted","Data":"f1d6aa56cf829c7d2df5a70620ed16198b8fb50ab8ba98ad8c6f4ea0268c865b"} Jul 21 19:48:06.715062 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:06.715015 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" podStartSLOduration=1.653386728 podStartE2EDuration="2.71500131s" totalImagesPullingTime="1.061614582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:48:06.713616016 +0000 UTC m=+649.898057725" watchObservedRunningTime="2026-07-21 19:48:06.71500131 +0000 UTC m=+649.899443039" Jul 21 19:48:07.291926 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.291888 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56"] Jul 21 19:48:07.295488 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.295467 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.298289 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.298270 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 21 19:48:07.306097 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.306075 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56"] Jul 21 19:48:07.396507 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.396465 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xrwr\" (UniqueName: \"kubernetes.io/projected/f5fe1b28-bcfb-499a-977d-d01614eb3b71-kube-api-access-9xrwr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.396703 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.396520 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.396703 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.396544 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.396703 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.396582 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.396703 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.396613 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fe1b28-bcfb-499a-977d-d01614eb3b71-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.396703 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.396667 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.497935 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.497901 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fe1b28-bcfb-499a-977d-d01614eb3b71-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.498104 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.497948 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.498104 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.498024 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xrwr\" (UniqueName: \"kubernetes.io/projected/f5fe1b28-bcfb-499a-977d-d01614eb3b71-kube-api-access-9xrwr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.498104 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.498061 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.498104 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.498081 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.498315 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.498157 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.498465 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.498446 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.498546 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.498500 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.498546 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.498526 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.500244 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.500223 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5fe1b28-bcfb-499a-977d-d01614eb3b71-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.500408 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.500393 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fe1b28-bcfb-499a-977d-d01614eb3b71-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.506453 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.506427 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xrwr\" (UniqueName: \"kubernetes.io/projected/f5fe1b28-bcfb-499a-977d-d01614eb3b71-kube-api-access-9xrwr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pwx56\" (UID: \"f5fe1b28-bcfb-499a-977d-d01614eb3b71\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.605302 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.605206 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:07.750939 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.750903 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56"] Jul 21 19:48:07.751754 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:48:07.751726 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5fe1b28_bcfb_499a_977d_d01614eb3b71.slice/crio-c4566a04035bba46f72694a80dad9adfada423c4e50e475917a9dc54d105e743 WatchSource:0}: Error finding container c4566a04035bba46f72694a80dad9adfada423c4e50e475917a9dc54d105e743: Status 404 returned error can't find the container with id c4566a04035bba46f72694a80dad9adfada423c4e50e475917a9dc54d105e743 Jul 21 19:48:07.792184 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:07.792153 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:48:08.708687 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:08.708645 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" event={"ID":"f5fe1b28-bcfb-499a-977d-d01614eb3b71","Type":"ContainerStarted","Data":"2d812a3e25e13cac4995880ec294ba7b1cd1cb33bf65d4c7db8f16acb485d119"} Jul 21 19:48:08.708855 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:08.708698 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" event={"ID":"f5fe1b28-bcfb-499a-977d-d01614eb3b71","Type":"ContainerStarted","Data":"c4566a04035bba46f72694a80dad9adfada423c4e50e475917a9dc54d105e743"} Jul 21 19:48:08.728306 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:08.728254 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" podStartSLOduration=1.475442763 podStartE2EDuration="1.728239037s" totalImagesPullingTime="252.796274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:48:08.726642406 +0000 UTC m=+651.911084108" watchObservedRunningTime="2026-07-21 19:48:08.728239037 +0000 UTC m=+651.912680718" Jul 21 19:48:15.020029 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:15.019991 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:15.020528 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:15.020075 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:15.032527 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:15.032501 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:15.759044 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:15.759014 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v5r5c" Jul 21 19:48:17.605948 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:17.605917 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:17.606349 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:17.606067 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:17.618632 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:17.618597 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:17.769857 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:17.769825 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pwx56" Jul 21 19:48:18.899795 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:48:18.899758 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:49:20.399465 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:49:20.399431 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:49:30.043292 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:49:30.043253 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:49:39.797619 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:49:39.797579 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:51:24.097343 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:51:24.097302 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:51:33.795607 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:51:33.795569 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:52:03.198091 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:52:03.198055 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:52:12.301070 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:52:12.300983 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:52:17.198181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:52:17.198148 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:52:17.199817 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:52:17.199791 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:52:21.396521 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:52:21.396478 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:52:30.295672 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:52:30.295630 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:52:40.205491 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:52:40.205456 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:52:49.899204 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:52:49.899172 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:53:01.502709 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:53:01.502677 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:53:11.202759 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:53:11.202722 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:53:17.302091 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:53:17.302051 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:53:25.002016 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:53:25.001978 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:53:37.003725 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:53:37.003684 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:53:48.797863 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:53:48.797782 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:54:12.694192 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:54:12.694157 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:54:22.594317 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:54:22.594283 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:54:38.403312 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:54:38.403272 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:54:48.193641 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:54:48.193607 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:54:57.906700 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:54:57.906662 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:55:06.801957 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:06.801921 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:55:16.195660 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:16.195575 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:55:43.031616 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:43.031574 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76"] Jul 21 19:55:43.032134 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:43.031777 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" podUID="8018e0b4-fc27-4eb0-80ea-2d516237118d" containerName="manager" containerID="cri-o://2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99" gracePeriod=10 Jul 21 19:55:45.992327 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:45.992305 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:55:46.082777 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.082696 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8018e0b4-fc27-4eb0-80ea-2d516237118d-extensions-socket-volume\") pod \"8018e0b4-fc27-4eb0-80ea-2d516237118d\" (UID: \"8018e0b4-fc27-4eb0-80ea-2d516237118d\") " Jul 21 19:55:46.082777 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.082739 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wphnc\" (UniqueName: \"kubernetes.io/projected/8018e0b4-fc27-4eb0-80ea-2d516237118d-kube-api-access-wphnc\") pod \"8018e0b4-fc27-4eb0-80ea-2d516237118d\" (UID: \"8018e0b4-fc27-4eb0-80ea-2d516237118d\") " Jul 21 19:55:46.083076 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.083054 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8018e0b4-fc27-4eb0-80ea-2d516237118d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8018e0b4-fc27-4eb0-80ea-2d516237118d" (UID: "8018e0b4-fc27-4eb0-80ea-2d516237118d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 19:55:46.084758 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.084729 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8018e0b4-fc27-4eb0-80ea-2d516237118d-kube-api-access-wphnc" (OuterVolumeSpecName: "kube-api-access-wphnc") pod "8018e0b4-fc27-4eb0-80ea-2d516237118d" (UID: "8018e0b4-fc27-4eb0-80ea-2d516237118d"). InnerVolumeSpecName "kube-api-access-wphnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 19:55:46.183522 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.183479 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8018e0b4-fc27-4eb0-80ea-2d516237118d-extensions-socket-volume\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:55:46.183522 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.183513 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wphnc\" (UniqueName: \"kubernetes.io/projected/8018e0b4-fc27-4eb0-80ea-2d516237118d-kube-api-access-wphnc\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 19:55:46.195291 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.195257 2557 generic.go:356] "Generic (PLEG): container finished" podID="8018e0b4-fc27-4eb0-80ea-2d516237118d" containerID="2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99" exitCode=0 Jul 21 19:55:46.195460 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.195300 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" event={"ID":"8018e0b4-fc27-4eb0-80ea-2d516237118d","Type":"ContainerDied","Data":"2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99"} Jul 21 19:55:46.195460 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.195309 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" Jul 21 19:55:46.195460 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.195343 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76" event={"ID":"8018e0b4-fc27-4eb0-80ea-2d516237118d","Type":"ContainerDied","Data":"52f7d4294fd344410fda8b3e83b100f8abdcce0f5d3a1e763f2402c598595a7b"} Jul 21 19:55:46.195460 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.195363 2557 scope.go:117] "RemoveContainer" containerID="2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99" Jul 21 19:55:46.206276 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.206260 2557 scope.go:117] "RemoveContainer" containerID="2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99" Jul 21 19:55:46.206532 ip-10-0-134-158 kubenswrapper[2557]: E0721 19:55:46.206512 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99\": container with ID starting with 2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99 not found: ID does not exist" containerID="2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99" Jul 21 19:55:46.206580 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.206542 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99"} err="failed to get container status \"2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99\": rpc error: code = NotFound desc = could not find container \"2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99\": container with ID starting with 2e9830e065b319d658faaf49f026e7abc59b9e318d7fd15589ee680d0c63ed99 not found: ID does not exist" Jul 21 19:55:46.217975 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.217948 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76"] Jul 21 19:55:46.221141 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.221118 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjs76"] Jul 21 19:55:46.505696 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:55:46.505660 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8018e0b4-fc27-4eb0-80ea-2d516237118d" path="/var/lib/kubelet/pods/8018e0b4-fc27-4eb0-80ea-2d516237118d/volumes" Jul 21 19:56:44.197704 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.197620 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85"] Jul 21 19:56:44.198144 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.198015 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8018e0b4-fc27-4eb0-80ea-2d516237118d" containerName="manager" Jul 21 19:56:44.198144 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.198028 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8018e0b4-fc27-4eb0-80ea-2d516237118d" containerName="manager" Jul 21 19:56:44.198144 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.198103 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8018e0b4-fc27-4eb0-80ea-2d516237118d" containerName="manager" Jul 21 19:56:44.202261 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.202245 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:56:44.205631 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.205611 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vrj9n\"" Jul 21 19:56:44.211806 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.211786 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85"] Jul 21 19:56:44.319168 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.319129 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnvkw\" (UniqueName: \"kubernetes.io/projected/e0875ea3-803b-46d5-9b59-c3d713d785a2-kube-api-access-wnvkw\") pod \"kuadrant-operator-controller-manager-86878c769f-mpl85\" (UID: \"e0875ea3-803b-46d5-9b59-c3d713d785a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:56:44.319355 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.319259 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0875ea3-803b-46d5-9b59-c3d713d785a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mpl85\" (UID: \"e0875ea3-803b-46d5-9b59-c3d713d785a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:56:44.420246 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.420209 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0875ea3-803b-46d5-9b59-c3d713d785a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mpl85\" (UID: \"e0875ea3-803b-46d5-9b59-c3d713d785a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:56:44.420446 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.420277 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnvkw\" (UniqueName: \"kubernetes.io/projected/e0875ea3-803b-46d5-9b59-c3d713d785a2-kube-api-access-wnvkw\") pod \"kuadrant-operator-controller-manager-86878c769f-mpl85\" (UID: \"e0875ea3-803b-46d5-9b59-c3d713d785a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:56:44.420618 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.420596 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0875ea3-803b-46d5-9b59-c3d713d785a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mpl85\" (UID: \"e0875ea3-803b-46d5-9b59-c3d713d785a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:56:44.429308 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.429275 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnvkw\" (UniqueName: \"kubernetes.io/projected/e0875ea3-803b-46d5-9b59-c3d713d785a2-kube-api-access-wnvkw\") pod \"kuadrant-operator-controller-manager-86878c769f-mpl85\" (UID: \"e0875ea3-803b-46d5-9b59-c3d713d785a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:56:44.513721 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.513644 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:56:44.656171 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:44.656141 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85"] Jul 21 19:56:44.657989 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:56:44.657964 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0875ea3_803b_46d5_9b59_c3d713d785a2.slice/crio-a64ab9b68d9a20c4517c56a4026146520ec10d1e76ba898826f6b1994ce113aa WatchSource:0}: Error finding container a64ab9b68d9a20c4517c56a4026146520ec10d1e76ba898826f6b1994ce113aa: Status 404 returned error can't find the container with id a64ab9b68d9a20c4517c56a4026146520ec10d1e76ba898826f6b1994ce113aa Jul 21 19:56:45.515727 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:45.515693 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" event={"ID":"e0875ea3-803b-46d5-9b59-c3d713d785a2","Type":"ContainerStarted","Data":"b5ba679f8aa7af47de10712e6ff20841621c64bc5d935ed8e07db56b48f1811a"} Jul 21 19:56:45.516116 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:45.515745 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" event={"ID":"e0875ea3-803b-46d5-9b59-c3d713d785a2","Type":"ContainerStarted","Data":"a64ab9b68d9a20c4517c56a4026146520ec10d1e76ba898826f6b1994ce113aa"} Jul 21 19:56:45.516116 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:45.515766 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:56:45.543517 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:45.543452 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" podStartSLOduration=1.543431115 podStartE2EDuration="1.543431115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:56:45.538640179 +0000 UTC m=+1168.723081881" watchObservedRunningTime="2026-07-21 19:56:45.543431115 +0000 UTC m=+1168.727872797" Jul 21 19:56:56.523976 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:56:56.523945 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mpl85" Jul 21 19:57:04.795129 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:57:04.795094 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:57:15.794181 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:57:15.794144 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:57:17.238248 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:57:17.238222 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:57:17.238699 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:57:17.238290 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 19:57:25.997587 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:57:25.997547 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:57:37.198369 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:57:37.198338 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:57:49.097943 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:57:49.097911 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:57:58.505216 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:57:58.505187 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:58:07.706529 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:58:07.706493 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:58:19.099852 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:58:19.099775 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:58:29.900127 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:58:29.900083 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:58:38.496021 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:58:38.495984 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:58:48.999369 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:58:48.999331 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:59:13.601266 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:13.601230 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:59:21.702743 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:21.702710 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:59:29.409904 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.409842 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt"] Jul 21 19:59:29.413484 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.413461 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.416085 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.416061 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-908c2342-kserve-self-signed-certs\"" Jul 21 19:59:29.433852 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.433821 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt"] Jul 21 19:59:29.447898 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.447848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d364867-e045-4013-aa37-bc865fdc7721-tls-certs\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.448083 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.447917 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-home\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.448083 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.447951 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npx5q\" (UniqueName: \"kubernetes.io/projected/1d364867-e045-4013-aa37-bc865fdc7721-kube-api-access-npx5q\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.448083 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.448042 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-dshm\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.448251 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.448166 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-model-cache\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.448251 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.448204 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-tmp-dir\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.548918 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.548864 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-home\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.548918 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.548916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npx5q\" (UniqueName: \"kubernetes.io/projected/1d364867-e045-4013-aa37-bc865fdc7721-kube-api-access-npx5q\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.549155 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.549051 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-dshm\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.549331 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.549293 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-home\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.549474 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.549434 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-model-cache\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.549474 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.549466 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-tmp-dir\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.549715 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.549621 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d364867-e045-4013-aa37-bc865fdc7721-tls-certs\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.549811 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.549793 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-model-cache\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.549937 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.549915 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-tmp-dir\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.551509 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.551488 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-dshm\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.551852 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.551832 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d364867-e045-4013-aa37-bc865fdc7721-tls-certs\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.558444 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.558422 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npx5q\" (UniqueName: \"kubernetes.io/projected/1d364867-e045-4013-aa37-bc865fdc7721-kube-api-access-npx5q\") pod \"e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.723114 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.723021 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:29.861561 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:29.861525 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt"] Jul 21 19:59:29.862690 ip-10-0-134-158 kubenswrapper[2557]: W0721 19:59:29.862659 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d364867_e045_4013_aa37_bc865fdc7721.slice/crio-3e0c7a51dcd335391e04ef0ef087f685f61c06b089be70c2ad42b5612741e5ca WatchSource:0}: Error finding container 3e0c7a51dcd335391e04ef0ef087f685f61c06b089be70c2ad42b5612741e5ca: Status 404 returned error can't find the container with id 3e0c7a51dcd335391e04ef0ef087f685f61c06b089be70c2ad42b5612741e5ca Jul 21 19:59:30.398555 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:30.398516 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" event={"ID":"1d364867-e045-4013-aa37-bc865fdc7721","Type":"ContainerStarted","Data":"05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4"} Jul 21 19:59:30.398734 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:30.398570 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" event={"ID":"1d364867-e045-4013-aa37-bc865fdc7721","Type":"ContainerStarted","Data":"3e0c7a51dcd335391e04ef0ef087f685f61c06b089be70c2ad42b5612741e5ca"} Jul 21 19:59:30.418661 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:30.418606 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" podStartSLOduration=1.418591457 podStartE2EDuration="1.418591457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 19:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 19:59:30.416703857 +0000 UTC m=+1333.601145538" watchObservedRunningTime="2026-07-21 19:59:30.418591457 +0000 UTC m=+1333.603033138" Jul 21 19:59:33.999826 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:33.999788 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 19:59:39.724046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:39.723952 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:39.724046 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:39.723996 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:39.736686 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:39.736660 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:40.464914 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:40.464862 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 19:59:47.300394 ip-10-0-134-158 kubenswrapper[2557]: I0721 19:59:47.300361 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:00:01.918073 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:01.918039 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt"] Jul 21 20:00:05.595276 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:05.595231 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:00:16.939705 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:16.939638 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" podUID="1d364867-e045-4013-aa37-bc865fdc7721" containerName="main" containerID="cri-o://05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4" gracePeriod=45 Jul 21 20:00:17.205613 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.205588 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 20:00:17.370371 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.370333 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npx5q\" (UniqueName: \"kubernetes.io/projected/1d364867-e045-4013-aa37-bc865fdc7721-kube-api-access-npx5q\") pod \"1d364867-e045-4013-aa37-bc865fdc7721\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " Jul 21 20:00:17.370597 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.370404 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d364867-e045-4013-aa37-bc865fdc7721-tls-certs\") pod \"1d364867-e045-4013-aa37-bc865fdc7721\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " Jul 21 20:00:17.370597 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.370457 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-home\") pod \"1d364867-e045-4013-aa37-bc865fdc7721\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " Jul 21 20:00:17.370597 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.370476 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-model-cache\") pod \"1d364867-e045-4013-aa37-bc865fdc7721\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " Jul 21 20:00:17.370597 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.370506 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-dshm\") pod \"1d364867-e045-4013-aa37-bc865fdc7721\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " Jul 21 20:00:17.370597 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.370555 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-tmp-dir\") pod \"1d364867-e045-4013-aa37-bc865fdc7721\" (UID: \"1d364867-e045-4013-aa37-bc865fdc7721\") " Jul 21 20:00:17.370906 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.370826 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d364867-e045-4013-aa37-bc865fdc7721" (UID: "1d364867-e045-4013-aa37-bc865fdc7721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:00:17.370906 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.370896 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-home" (OuterVolumeSpecName: "home") pod "1d364867-e045-4013-aa37-bc865fdc7721" (UID: "1d364867-e045-4013-aa37-bc865fdc7721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:00:17.371074 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.371043 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d364867-e045-4013-aa37-bc865fdc7721" (UID: "1d364867-e045-4013-aa37-bc865fdc7721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:00:17.373142 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.373120 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d364867-e045-4013-aa37-bc865fdc7721-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d364867-e045-4013-aa37-bc865fdc7721" (UID: "1d364867-e045-4013-aa37-bc865fdc7721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:00:17.373237 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.373204 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-dshm" (OuterVolumeSpecName: "dshm") pod "1d364867-e045-4013-aa37-bc865fdc7721" (UID: "1d364867-e045-4013-aa37-bc865fdc7721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:00:17.373292 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.373238 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d364867-e045-4013-aa37-bc865fdc7721-kube-api-access-npx5q" (OuterVolumeSpecName: "kube-api-access-npx5q") pod "1d364867-e045-4013-aa37-bc865fdc7721" (UID: "1d364867-e045-4013-aa37-bc865fdc7721"). InnerVolumeSpecName "kube-api-access-npx5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:00:17.472130 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.472035 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-tmp-dir\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:00:17.472130 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.472062 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npx5q\" (UniqueName: \"kubernetes.io/projected/1d364867-e045-4013-aa37-bc865fdc7721-kube-api-access-npx5q\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:00:17.472130 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.472071 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d364867-e045-4013-aa37-bc865fdc7721-tls-certs\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:00:17.472130 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.472080 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-home\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:00:17.472130 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.472089 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-model-cache\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:00:17.472130 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.472096 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d364867-e045-4013-aa37-bc865fdc7721-dshm\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:00:17.654844 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.654807 2557 generic.go:356] "Generic (PLEG): container finished" podID="1d364867-e045-4013-aa37-bc865fdc7721" containerID="05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4" exitCode=0 Jul 21 20:00:17.655068 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.654861 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" Jul 21 20:00:17.655068 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.654906 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" event={"ID":"1d364867-e045-4013-aa37-bc865fdc7721","Type":"ContainerDied","Data":"05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4"} Jul 21 20:00:17.655068 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.654947 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt" event={"ID":"1d364867-e045-4013-aa37-bc865fdc7721","Type":"ContainerDied","Data":"3e0c7a51dcd335391e04ef0ef087f685f61c06b089be70c2ad42b5612741e5ca"} Jul 21 20:00:17.655068 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.654966 2557 scope.go:117] "RemoveContainer" containerID="05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4" Jul 21 20:00:17.664838 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.664818 2557 scope.go:117] "RemoveContainer" containerID="05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4" Jul 21 20:00:17.665084 ip-10-0-134-158 kubenswrapper[2557]: E0721 20:00:17.665065 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4\": container with ID starting with 05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4 not found: ID does not exist" containerID="05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4" Jul 21 20:00:17.665157 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.665093 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4"} err="failed to get container status \"05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4\": rpc error: code = NotFound desc = could not find container \"05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4\": container with ID starting with 05c61a96f1e4cfee2128db82fc0f6be2855b39681bd97c9740272668115ab5e4 not found: ID does not exist" Jul 21 20:00:17.676581 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.676555 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt"] Jul 21 20:00:17.680769 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:17.680746 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-908c2342-kserve-7bfffbfff-dlbnt"] Jul 21 20:00:18.506457 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:18.506421 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d364867-e045-4013-aa37-bc865fdc7721" path="/var/lib/kubelet/pods/1d364867-e045-4013-aa37-bc865fdc7721/volumes" Jul 21 20:00:19.993575 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:19.993540 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:00:27.000315 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:27.000268 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:00:42.506244 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:42.506209 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:00:54.194947 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:00:54.194910 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:01:03.301327 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:01:03.301289 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:01:14.005820 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:01:14.005742 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:01:23.594301 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:01:23.594270 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:01:32.701602 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:01:32.701564 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:01:45.802699 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:01:45.802655 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:01:55.295258 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:01:55.295223 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:02:04.099733 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:02:04.099699 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:02:16.201361 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:02:16.201324 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:02:17.281317 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:02:17.281285 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:02:17.282821 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:02:17.282803 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:02:27.092759 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:02:27.092721 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:02:36.596741 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:02:36.596704 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:02:48.997198 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:02:48.997124 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:03:01.402722 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:01.402684 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:03:14.498091 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:14.498053 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:03:23.402616 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:23.402577 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:03:45.695020 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:45.694973 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:03:50.901708 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:50.901671 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m"] Jul 21 20:03:50.902089 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:50.902075 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d364867-e045-4013-aa37-bc865fdc7721" containerName="main" Jul 21 20:03:50.902135 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:50.902090 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d364867-e045-4013-aa37-bc865fdc7721" containerName="main" Jul 21 20:03:50.902170 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:50.902151 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d364867-e045-4013-aa37-bc865fdc7721" containerName="main" Jul 21 20:03:50.905156 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:50.905135 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:50.908578 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:50.908556 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fd2497fa-openshift-default-dockercfg-ttsjj\"" Jul 21 20:03:50.964829 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:50.964786 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m"] Jul 21 20:03:51.060017 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.059984 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-data\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.060188 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.060033 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-credential-socket\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.060188 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.060051 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106336b3-aa66-44b1-88ab-810c4e443b60-istiod-ca-cert\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.060188 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.060071 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjr7\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-kube-api-access-vkjr7\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.060188 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.060101 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-socket\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.060188 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.060123 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-istio-token\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.060358 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.060203 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-envoy\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.060358 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.060239 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106336b3-aa66-44b1-88ab-810c4e443b60-istio-podinfo\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.060358 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.060271 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-certs\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.161664 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.161574 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-credential-socket\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.161664 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.161612 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106336b3-aa66-44b1-88ab-810c4e443b60-istiod-ca-cert\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.161664 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.161633 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjr7\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-kube-api-access-vkjr7\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.161938 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.161912 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-socket\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.161985 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.161956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-istio-token\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.162024 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.162001 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-credential-socket\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.162070 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.162057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-envoy\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.162116 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.162101 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106336b3-aa66-44b1-88ab-810c4e443b60-istio-podinfo\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.162169 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.162155 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-certs\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.162235 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.162222 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-data\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.162296 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.162233 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-socket\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.162439 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.162421 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106336b3-aa66-44b1-88ab-810c4e443b60-istiod-ca-cert\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.162567 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.162536 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-certs\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.162567 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.162549 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-data\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.164325 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.164304 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-envoy\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.164463 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.164445 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106336b3-aa66-44b1-88ab-810c4e443b60-istio-podinfo\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.171519 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.171491 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-istio-token\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.171627 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.171609 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjr7\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-kube-api-access-vkjr7\") pod \"e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.215205 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.215171 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:51.359181 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.359156 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m"] Jul 21 20:03:51.361213 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:03:51.361186 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106336b3_aa66_44b1_88ab_810c4e443b60.slice/crio-a0cdefc53100197da8a7499f2900187c1b1205f487320567b16e5f14450e66ec WatchSource:0}: Error finding container a0cdefc53100197da8a7499f2900187c1b1205f487320567b16e5f14450e66ec: Status 404 returned error can't find the container with id a0cdefc53100197da8a7499f2900187c1b1205f487320567b16e5f14450e66ec Jul 21 20:03:51.363276 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.363245 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 20:03:51.363399 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.363312 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 20:03:51.363399 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.363339 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 20:03:51.807573 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.807482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" event={"ID":"106336b3-aa66-44b1-88ab-810c4e443b60","Type":"ContainerStarted","Data":"5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff"} Jul 21 20:03:51.807573 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.807532 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" event={"ID":"106336b3-aa66-44b1-88ab-810c4e443b60","Type":"ContainerStarted","Data":"a0cdefc53100197da8a7499f2900187c1b1205f487320567b16e5f14450e66ec"} Jul 21 20:03:51.828940 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:51.828846 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" podStartSLOduration=1.8288312580000001 podStartE2EDuration="1.828831258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:03:51.827131143 +0000 UTC m=+1595.011572854" watchObservedRunningTime="2026-07-21 20:03:51.828831258 +0000 UTC m=+1595.013272917" Jul 21 20:03:52.215695 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:52.215663 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:52.219930 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:52.219904 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:52.812820 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:52.812786 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:52.813827 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:52.813810 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:03:56.289495 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:03:56.289456 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:04:04.891435 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:04.891400 2557 generic.go:356] "Generic (PLEG): container finished" podID="106336b3-aa66-44b1-88ab-810c4e443b60" containerID="5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff" exitCode=0 Jul 21 20:04:04.891847 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:04.891475 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" event={"ID":"106336b3-aa66-44b1-88ab-810c4e443b60","Type":"ContainerDied","Data":"5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff"} Jul 21 20:04:04.891933 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:04.891903 2557 scope.go:117] "RemoveContainer" containerID="5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff" Jul 21 20:04:04.892938 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:04.892907 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 20:04:04.893051 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:04.892975 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 20:04:04.893051 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:04.893004 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 20:04:05.898742 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:05.898706 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" event={"ID":"106336b3-aa66-44b1-88ab-810c4e443b60","Type":"ContainerStarted","Data":"a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640"} Jul 21 20:04:06.215576 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:06.215489 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:04:06.216606 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:06.216585 2557 patch_prober.go:28] interesting pod/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 20:04:06.216717 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:06.216637 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 20:04:07.215868 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:07.215834 2557 patch_prober.go:28] interesting pod/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 20:04:07.216260 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:07.215918 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 20:04:07.813464 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:07.813431 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:04:08.215754 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:08.215719 2557 patch_prober.go:28] interesting pod/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 20:04:08.215961 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:08.215771 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 20:04:09.215538 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:09.215501 2557 patch_prober.go:28] interesting pod/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 20:04:09.215720 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:09.215557 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 20:04:10.219520 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:10.219480 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:04:10.220513 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:10.220493 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:04:32.101309 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:32.101270 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:04:34.048545 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:34.048508 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m"] Jul 21 20:04:34.049030 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:34.048732 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" containerID="cri-o://a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640" gracePeriod=30 Jul 21 20:04:39.310505 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.310478 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:04:39.405765 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.405732 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-envoy\") pod \"106336b3-aa66-44b1-88ab-810c4e443b60\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " Jul 21 20:04:39.405987 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.405816 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106336b3-aa66-44b1-88ab-810c4e443b60-istiod-ca-cert\") pod \"106336b3-aa66-44b1-88ab-810c4e443b60\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " Jul 21 20:04:39.405987 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.405844 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-data\") pod \"106336b3-aa66-44b1-88ab-810c4e443b60\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " Jul 21 20:04:39.405987 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.405868 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-socket\") pod \"106336b3-aa66-44b1-88ab-810c4e443b60\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " Jul 21 20:04:39.406168 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406038 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106336b3-aa66-44b1-88ab-810c4e443b60-istio-podinfo\") pod \"106336b3-aa66-44b1-88ab-810c4e443b60\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " Jul 21 20:04:39.406168 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406081 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkjr7\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-kube-api-access-vkjr7\") pod \"106336b3-aa66-44b1-88ab-810c4e443b60\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " Jul 21 20:04:39.406168 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406106 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-istio-token\") pod \"106336b3-aa66-44b1-88ab-810c4e443b60\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " Jul 21 20:04:39.406168 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406137 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-credential-socket\") pod \"106336b3-aa66-44b1-88ab-810c4e443b60\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " Jul 21 20:04:39.406423 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406214 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "106336b3-aa66-44b1-88ab-810c4e443b60" (UID: "106336b3-aa66-44b1-88ab-810c4e443b60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:04:39.406423 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406336 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-certs\") pod \"106336b3-aa66-44b1-88ab-810c4e443b60\" (UID: \"106336b3-aa66-44b1-88ab-810c4e443b60\") " Jul 21 20:04:39.406423 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406362 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "106336b3-aa66-44b1-88ab-810c4e443b60" (UID: "106336b3-aa66-44b1-88ab-810c4e443b60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:04:39.406666 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406641 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "106336b3-aa66-44b1-88ab-810c4e443b60" (UID: "106336b3-aa66-44b1-88ab-810c4e443b60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:04:39.406740 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406689 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106336b3-aa66-44b1-88ab-810c4e443b60-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "106336b3-aa66-44b1-88ab-810c4e443b60" (UID: "106336b3-aa66-44b1-88ab-810c4e443b60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:04:39.406740 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406693 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-data" (OuterVolumeSpecName: "istio-data") pod "106336b3-aa66-44b1-88ab-810c4e443b60" (UID: "106336b3-aa66-44b1-88ab-810c4e443b60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:04:39.406847 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406810 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106336b3-aa66-44b1-88ab-810c4e443b60-istiod-ca-cert\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:04:39.406847 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406826 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-data\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:04:39.406847 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406838 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-socket\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:04:39.407049 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406853 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-credential-socket\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:04:39.407049 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.406866 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-workload-certs\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:04:39.408114 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.408089 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "106336b3-aa66-44b1-88ab-810c4e443b60" (UID: "106336b3-aa66-44b1-88ab-810c4e443b60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:04:39.408239 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.408194 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/106336b3-aa66-44b1-88ab-810c4e443b60-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "106336b3-aa66-44b1-88ab-810c4e443b60" (UID: "106336b3-aa66-44b1-88ab-810c4e443b60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 20:04:39.408293 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.408275 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-istio-token" (OuterVolumeSpecName: "istio-token") pod "106336b3-aa66-44b1-88ab-810c4e443b60" (UID: "106336b3-aa66-44b1-88ab-810c4e443b60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:04:39.408543 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.408523 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-kube-api-access-vkjr7" (OuterVolumeSpecName: "kube-api-access-vkjr7") pod "106336b3-aa66-44b1-88ab-810c4e443b60" (UID: "106336b3-aa66-44b1-88ab-810c4e443b60"). InnerVolumeSpecName "kube-api-access-vkjr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:04:39.508119 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.508086 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106336b3-aa66-44b1-88ab-810c4e443b60-istio-envoy\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:04:39.508119 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.508117 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106336b3-aa66-44b1-88ab-810c4e443b60-istio-podinfo\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:04:39.508329 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.508129 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjr7\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-kube-api-access-vkjr7\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:04:39.508329 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:39.508143 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106336b3-aa66-44b1-88ab-810c4e443b60-istio-token\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:04:40.089358 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.089324 2557 generic.go:356] "Generic (PLEG): container finished" podID="106336b3-aa66-44b1-88ab-810c4e443b60" containerID="a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640" exitCode=0 Jul 21 20:04:40.089506 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.089364 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" event={"ID":"106336b3-aa66-44b1-88ab-810c4e443b60","Type":"ContainerDied","Data":"a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640"} Jul 21 20:04:40.089506 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.089369 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" Jul 21 20:04:40.089506 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.089408 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m" event={"ID":"106336b3-aa66-44b1-88ab-810c4e443b60","Type":"ContainerDied","Data":"a0cdefc53100197da8a7499f2900187c1b1205f487320567b16e5f14450e66ec"} Jul 21 20:04:40.089506 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.089429 2557 scope.go:117] "RemoveContainer" containerID="a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640" Jul 21 20:04:40.098993 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.098979 2557 scope.go:117] "RemoveContainer" containerID="5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff" Jul 21 20:04:40.107277 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.107260 2557 scope.go:117] "RemoveContainer" containerID="a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640" Jul 21 20:04:40.107506 ip-10-0-134-158 kubenswrapper[2557]: E0721 20:04:40.107490 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640\": container with ID starting with a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640 not found: ID does not exist" containerID="a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640" Jul 21 20:04:40.107558 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.107513 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640"} err="failed to get container status \"a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640\": rpc error: code = NotFound desc = could not find container \"a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640\": container with ID starting with a7064e58059564a3997f8998d1e6f23c3c55ed6cc41469806d1d80984711d640 not found: ID does not exist" Jul 21 20:04:40.107558 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.107535 2557 scope.go:117] "RemoveContainer" containerID="5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff" Jul 21 20:04:40.107713 ip-10-0-134-158 kubenswrapper[2557]: E0721 20:04:40.107698 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff\": container with ID starting with 5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff not found: ID does not exist" containerID="5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff" Jul 21 20:04:40.107757 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.107717 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff"} err="failed to get container status \"5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff\": rpc error: code = NotFound desc = could not find container \"5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff\": container with ID starting with 5c5af4625e08937e719c3b9ec009bc27fa69665ece04f4e60f4069fe0e934aff not found: ID does not exist" Jul 21 20:04:40.112670 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.112648 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m"] Jul 21 20:04:40.119911 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.119868 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fd2497fa-openshift-default-6785f9874f-nr99m"] Jul 21 20:04:40.505059 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:40.505018 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" path="/var/lib/kubelet/pods/106336b3-aa66-44b1-88ab-810c4e443b60/volumes" Jul 21 20:04:44.196281 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:04:44.196240 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:05:19.500185 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:05:19.500143 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:05:30.797813 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:05:30.797757 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:06:03.498224 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:06:03.498187 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:06:14.990118 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:06:14.990082 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:06:47.495600 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:06:47.495569 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:07:17.319206 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:07:17.319170 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:07:17.321317 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:07:17.321294 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:08:14.810526 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:08:14.810490 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:08:25.299721 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:08:25.299687 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:08:35.702964 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:08:35.702926 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:08:43.904138 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:08:43.904043 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:08:59.989365 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:08:59.989324 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:09:37.601409 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:37.601372 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:09:48.200805 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.200762 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d"] Jul 21 20:09:48.201194 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.201180 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" Jul 21 20:09:48.201239 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.201195 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" Jul 21 20:09:48.201239 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.201212 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" Jul 21 20:09:48.201239 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.201218 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" Jul 21 20:09:48.201331 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.201285 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" Jul 21 20:09:48.201331 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.201292 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="106336b3-aa66-44b1-88ab-810c4e443b60" containerName="istio-proxy" Jul 21 20:09:48.204366 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.204347 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.207189 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.207165 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-78cc8875\"/\"kube-root-ca.crt\"" Jul 21 20:09:48.208355 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.208329 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-78cc8875\"/\"default-dockercfg-gpnl7\"" Jul 21 20:09:48.208479 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.208329 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-78cc8875\"/\"openshift-service-ca.crt\"" Jul 21 20:09:48.208549 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.208496 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-78cc8875\"/\"e2e-lifecycle-model-78cc8875-kserve-self-signed-certs\"" Jul 21 20:09:48.215248 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.215223 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d"] Jul 21 20:09:48.277002 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.276973 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-dshm\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.277002 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.277007 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tmp-dir\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.277228 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.277106 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-model-cache\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.277228 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.277146 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-home\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.277228 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.277224 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tls-certs\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.277351 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.277254 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stcx8\" (UniqueName: \"kubernetes.io/projected/e0fb68f3-744a-49db-b400-b3f5abb65b3a-kube-api-access-stcx8\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.378691 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.378653 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-model-cache\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.378926 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.378709 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-home\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.378926 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.378770 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tls-certs\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.378926 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.378816 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stcx8\" (UniqueName: \"kubernetes.io/projected/e0fb68f3-744a-49db-b400-b3f5abb65b3a-kube-api-access-stcx8\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.378926 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.378850 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-dshm\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.379217 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.378996 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tmp-dir\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.379217 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.379154 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-home\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.379318 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.379242 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-model-cache\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.379364 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.379340 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tmp-dir\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.381236 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.381201 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-dshm\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.381358 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.381310 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tls-certs\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.390438 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.390412 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stcx8\" (UniqueName: \"kubernetes.io/projected/e0fb68f3-744a-49db-b400-b3f5abb65b3a-kube-api-access-stcx8\") pod \"e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.514422 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.514342 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:48.664233 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.664205 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d"] Jul 21 20:09:48.667009 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:09:48.666973 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0fb68f3_744a_49db_b400_b3f5abb65b3a.slice/crio-8cca151d33dee0c831502a95747ae165566e2310dbf94847cb5820b1d8b22c0b WatchSource:0}: Error finding container 8cca151d33dee0c831502a95747ae165566e2310dbf94847cb5820b1d8b22c0b: Status 404 returned error can't find the container with id 8cca151d33dee0c831502a95747ae165566e2310dbf94847cb5820b1d8b22c0b Jul 21 20:09:48.755738 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.755702 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" event={"ID":"e0fb68f3-744a-49db-b400-b3f5abb65b3a","Type":"ContainerStarted","Data":"4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a"} Jul 21 20:09:48.755929 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.755749 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" event={"ID":"e0fb68f3-744a-49db-b400-b3f5abb65b3a","Type":"ContainerStarted","Data":"8cca151d33dee0c831502a95747ae165566e2310dbf94847cb5820b1d8b22c0b"} Jul 21 20:09:48.773535 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:48.773423 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" podStartSLOduration=0.773402561 podStartE2EDuration="773.402561ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:09:48.771519226 +0000 UTC m=+1951.955960944" watchObservedRunningTime="2026-07-21 20:09:48.773402561 +0000 UTC m=+1951.957844247" Jul 21 20:09:51.296738 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:51.296703 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:09:58.514832 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:58.514796 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:58.515232 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:58.514844 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:58.527469 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:58.527441 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:09:58.823575 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:09:58.823494 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:10:14.806114 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:10:14.806036 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:10:33.097150 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:10:33.097113 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:10:43.895521 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:10:43.895483 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:10:46.530916 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:10:46.530859 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d"] Jul 21 20:11:01.552777 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.552711 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" podUID="e0fb68f3-744a-49db-b400-b3f5abb65b3a" containerName="main" containerID="cri-o://4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a" gracePeriod=45 Jul 21 20:11:01.808886 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.808817 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:11:01.970123 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970084 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-dshm\") pod \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " Jul 21 20:11:01.970302 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970133 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-home\") pod \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " Jul 21 20:11:01.970302 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970152 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stcx8\" (UniqueName: \"kubernetes.io/projected/e0fb68f3-744a-49db-b400-b3f5abb65b3a-kube-api-access-stcx8\") pod \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " Jul 21 20:11:01.970302 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970169 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tls-certs\") pod \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " Jul 21 20:11:01.970302 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970224 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-model-cache\") pod \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " Jul 21 20:11:01.970302 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970263 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tmp-dir\") pod \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\" (UID: \"e0fb68f3-744a-49db-b400-b3f5abb65b3a\") " Jul 21 20:11:01.970551 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970443 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-home" (OuterVolumeSpecName: "home") pod "e0fb68f3-744a-49db-b400-b3f5abb65b3a" (UID: "e0fb68f3-744a-49db-b400-b3f5abb65b3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:11:01.970612 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970557 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0fb68f3-744a-49db-b400-b3f5abb65b3a" (UID: "e0fb68f3-744a-49db-b400-b3f5abb65b3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:11:01.970673 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970623 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-home\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:11:01.970673 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970637 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-model-cache\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:11:01.970673 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.970635 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0fb68f3-744a-49db-b400-b3f5abb65b3a" (UID: "e0fb68f3-744a-49db-b400-b3f5abb65b3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:11:01.972500 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.972468 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0fb68f3-744a-49db-b400-b3f5abb65b3a" (UID: "e0fb68f3-744a-49db-b400-b3f5abb65b3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:11:01.972620 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.972512 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fb68f3-744a-49db-b400-b3f5abb65b3a-kube-api-access-stcx8" (OuterVolumeSpecName: "kube-api-access-stcx8") pod "e0fb68f3-744a-49db-b400-b3f5abb65b3a" (UID: "e0fb68f3-744a-49db-b400-b3f5abb65b3a"). InnerVolumeSpecName "kube-api-access-stcx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:11:01.972620 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:01.972559 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-dshm" (OuterVolumeSpecName: "dshm") pod "e0fb68f3-744a-49db-b400-b3f5abb65b3a" (UID: "e0fb68f3-744a-49db-b400-b3f5abb65b3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:11:02.071949 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.071826 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stcx8\" (UniqueName: \"kubernetes.io/projected/e0fb68f3-744a-49db-b400-b3f5abb65b3a-kube-api-access-stcx8\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:11:02.071949 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.071856 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tls-certs\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:11:02.071949 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.071869 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-tmp-dir\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:11:02.071949 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.071909 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fb68f3-744a-49db-b400-b3f5abb65b3a-dshm\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:11:02.161662 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.161627 2557 generic.go:356] "Generic (PLEG): container finished" podID="e0fb68f3-744a-49db-b400-b3f5abb65b3a" containerID="4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a" exitCode=0 Jul 21 20:11:02.161828 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.161675 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" Jul 21 20:11:02.161828 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.161712 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" event={"ID":"e0fb68f3-744a-49db-b400-b3f5abb65b3a","Type":"ContainerDied","Data":"4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a"} Jul 21 20:11:02.161828 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.161764 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d" event={"ID":"e0fb68f3-744a-49db-b400-b3f5abb65b3a","Type":"ContainerDied","Data":"8cca151d33dee0c831502a95747ae165566e2310dbf94847cb5820b1d8b22c0b"} Jul 21 20:11:02.161828 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.161777 2557 scope.go:117] "RemoveContainer" containerID="4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a" Jul 21 20:11:02.171123 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.171104 2557 scope.go:117] "RemoveContainer" containerID="4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a" Jul 21 20:11:02.171373 ip-10-0-134-158 kubenswrapper[2557]: E0721 20:11:02.171356 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a\": container with ID starting with 4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a not found: ID does not exist" containerID="4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a" Jul 21 20:11:02.171444 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.171381 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a"} err="failed to get container status \"4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a\": rpc error: code = NotFound desc = could not find container \"4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a\": container with ID starting with 4e3e1b5a5c2b89c77b3530b500f02358e489fe8b2e3c99ffa2798a1c0eed258a not found: ID does not exist" Jul 21 20:11:02.182441 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.182403 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d"] Jul 21 20:11:02.185816 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.185791 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-78cc8875/e2e-lifecycle-model-78cc8875-kserve-779c8cb667-lxq5d"] Jul 21 20:11:02.505712 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:02.505675 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0fb68f3-744a-49db-b400-b3f5abb65b3a" path="/var/lib/kubelet/pods/e0fb68f3-744a-49db-b400-b3f5abb65b3a/volumes" Jul 21 20:11:49.700323 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:49.700242 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:11:59.098414 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:11:59.098375 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:12:17.353504 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:12:17.353477 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:12:17.356051 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:12:17.356027 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:12:47.971272 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:12:47.971227 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:12:56.914427 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:12:56.914384 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:13:03.702359 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.702325 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv"] Jul 21 20:13:03.702916 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.702897 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fb68f3-744a-49db-b400-b3f5abb65b3a" containerName="main" Jul 21 20:13:03.703010 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.702919 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fb68f3-744a-49db-b400-b3f5abb65b3a" containerName="main" Jul 21 20:13:03.703066 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.703044 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0fb68f3-744a-49db-b400-b3f5abb65b3a" containerName="main" Jul 21 20:13:03.706119 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.706089 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.710061 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.710027 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c3ae20\"/\"openshift-service-ca.crt\"" Jul 21 20:13:03.710197 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.710088 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c3ae20\"/\"test-model-c3ae20-kserve-self-signed-certs\"" Jul 21 20:13:03.710197 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.710128 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c3ae20\"/\"default-dockercfg-t82wb\"" Jul 21 20:13:03.710197 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.710128 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c3ae20\"/\"kube-root-ca.crt\"" Jul 21 20:13:03.714568 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.714542 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv"] Jul 21 20:13:03.803536 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.803507 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-dshm\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.803713 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.803553 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-model-cache\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.803713 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.803582 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lvp\" (UniqueName: \"kubernetes.io/projected/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-kube-api-access-b4lvp\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.803713 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.803682 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tls-certs\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.803842 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.803733 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tmp-dir\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.803842 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.803758 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-home\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.904308 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.904273 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-dshm\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.907897 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.904686 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-model-cache\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.907897 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.904759 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lvp\" (UniqueName: \"kubernetes.io/projected/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-kube-api-access-b4lvp\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.907897 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.905084 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-model-cache\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.907897 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.905245 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tls-certs\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.907897 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.905311 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tmp-dir\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.907897 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.905350 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-home\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.907897 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.905732 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-home\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.907897 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.907547 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-dshm\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.907897 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.907814 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tmp-dir\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.912447 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.912417 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tls-certs\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:03.922370 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:03.922342 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lvp\" (UniqueName: \"kubernetes.io/projected/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-kube-api-access-b4lvp\") pod \"test-model-c3ae20-kserve-79f56f4dfc-6q2wv\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:04.016585 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:04.016552 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:04.161693 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:04.161669 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv"] Jul 21 20:13:04.163004 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:13:04.162977 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e4e038a_c4c9_4e5d_a1dd_d32792a2b636.slice/crio-c00ddc52c59da6845a9f9c4bf3547ac4e70877865de1b72b98c85e0c034c7fec WatchSource:0}: Error finding container c00ddc52c59da6845a9f9c4bf3547ac4e70877865de1b72b98c85e0c034c7fec: Status 404 returned error can't find the container with id c00ddc52c59da6845a9f9c4bf3547ac4e70877865de1b72b98c85e0c034c7fec Jul 21 20:13:04.810211 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:04.810171 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" event={"ID":"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636","Type":"ContainerStarted","Data":"c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258"} Jul 21 20:13:04.810211 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:04.810218 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" event={"ID":"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636","Type":"ContainerStarted","Data":"c00ddc52c59da6845a9f9c4bf3547ac4e70877865de1b72b98c85e0c034c7fec"} Jul 21 20:13:04.829709 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:04.829662 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" podStartSLOduration=1.829648306 podStartE2EDuration="1.829648306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:13:04.827992917 +0000 UTC m=+2148.012434601" watchObservedRunningTime="2026-07-21 20:13:04.829648306 +0000 UTC m=+2148.014089986" Jul 21 20:13:09.108118 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:09.108084 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:13:14.017614 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:14.017576 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:14.018067 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:14.017796 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:14.030418 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:14.030386 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:14.872286 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:14.872259 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:13:49.206085 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:13:49.206037 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:14:01.608747 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:01.608707 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:14:25.438905 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:25.438857 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv"] Jul 21 20:14:30.325511 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:30.325474 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:14:40.459518 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.459406 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" podUID="4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" containerName="main" containerID="cri-o://c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258" gracePeriod=45 Jul 21 20:14:40.717291 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.717225 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:14:40.727889 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.727844 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-model-cache\") pod \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " Jul 21 20:14:40.728008 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.727901 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tmp-dir\") pod \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " Jul 21 20:14:40.728008 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.727972 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tls-certs\") pod \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " Jul 21 20:14:40.728008 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.727998 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4lvp\" (UniqueName: \"kubernetes.io/projected/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-kube-api-access-b4lvp\") pod \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " Jul 21 20:14:40.728181 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.728025 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-home\") pod \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " Jul 21 20:14:40.728181 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.728052 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-dshm\") pod \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\" (UID: \"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636\") " Jul 21 20:14:40.728181 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.728051 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" (UID: "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:14:40.728312 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.728181 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" (UID: "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:14:40.728312 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.728282 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-home" (OuterVolumeSpecName: "home") pod "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" (UID: "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:14:40.728389 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.728333 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-model-cache\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:14:40.728389 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.728344 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tmp-dir\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:14:40.728389 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.728352 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-home\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:14:40.730203 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.730170 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-kube-api-access-b4lvp" (OuterVolumeSpecName: "kube-api-access-b4lvp") pod "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" (UID: "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636"). InnerVolumeSpecName "kube-api-access-b4lvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:14:40.730203 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.730183 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" (UID: "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:14:40.730344 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.730183 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-dshm" (OuterVolumeSpecName: "dshm") pod "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" (UID: "4e4e038a-c4c9-4e5d-a1dd-d32792a2b636"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:14:40.829142 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.829111 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-dshm\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:14:40.829142 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.829137 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-tls-certs\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:14:40.829142 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:40.829147 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4lvp\" (UniqueName: \"kubernetes.io/projected/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636-kube-api-access-b4lvp\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:14:41.328971 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:41.328939 2557 generic.go:356] "Generic (PLEG): container finished" podID="4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" containerID="c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258" exitCode=0 Jul 21 20:14:41.329140 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:41.328982 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" Jul 21 20:14:41.329140 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:41.328979 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" event={"ID":"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636","Type":"ContainerDied","Data":"c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258"} Jul 21 20:14:41.329140 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:41.329088 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv" event={"ID":"4e4e038a-c4c9-4e5d-a1dd-d32792a2b636","Type":"ContainerDied","Data":"c00ddc52c59da6845a9f9c4bf3547ac4e70877865de1b72b98c85e0c034c7fec"} Jul 21 20:14:41.329140 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:41.329106 2557 scope.go:117] "RemoveContainer" containerID="c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258" Jul 21 20:14:41.339204 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:41.339182 2557 scope.go:117] "RemoveContainer" containerID="c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258" Jul 21 20:14:41.339516 ip-10-0-134-158 kubenswrapper[2557]: E0721 20:14:41.339480 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258\": container with ID starting with c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258 not found: ID does not exist" containerID="c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258" Jul 21 20:14:41.339581 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:41.339544 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258"} err="failed to get container status \"c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258\": rpc error: code = NotFound desc = could not find container \"c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258\": container with ID starting with c3f4417cd3c2c6d03c4cba303292db003825907e70fc335e9f971cac05d98258 not found: ID does not exist" Jul 21 20:14:41.354802 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:41.354778 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv"] Jul 21 20:14:41.356882 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:41.356851 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c3ae20/test-model-c3ae20-kserve-79f56f4dfc-6q2wv"] Jul 21 20:14:42.505478 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:42.505442 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" path="/var/lib/kubelet/pods/4e4e038a-c4c9-4e5d-a1dd-d32792a2b636/volumes" Jul 21 20:14:42.845778 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:14:42.845700 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:15:03.618913 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:15:03.618859 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:15:22.159547 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:15:22.159508 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:15:33.156193 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:15:33.156154 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:15:56.315379 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:15:56.315344 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:16:26.415930 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:16:26.415895 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:16:41.311003 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:16:41.310965 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:16:52.511657 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:16:52.511620 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:17:05.409627 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.409594 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz"] Jul 21 20:17:05.410063 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.410049 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" containerName="main" Jul 21 20:17:05.410120 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.410065 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" containerName="main" Jul 21 20:17:05.410160 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.410136 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4e038a-c4c9-4e5d-a1dd-d32792a2b636" containerName="main" Jul 21 20:17:05.413074 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.413055 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.417561 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.417529 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-065403\"/\"auth-test-model-065403-kserve-self-signed-certs\"" Jul 21 20:17:05.417691 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.417559 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-065403\"/\"kube-root-ca.crt\"" Jul 21 20:17:05.417691 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.417566 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-065403\"/\"openshift-service-ca.crt\"" Jul 21 20:17:05.417691 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.417566 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-065403\"/\"default-dockercfg-2qvt5\"" Jul 21 20:17:05.424796 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.424768 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz"] Jul 21 20:17:05.469526 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.469490 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjdxx\" (UniqueName: \"kubernetes.io/projected/69f80b73-c836-46b4-a3b7-1d72d3152714-kube-api-access-cjdxx\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.469726 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.469579 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-tmp-dir\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.469726 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.469604 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f80b73-c836-46b4-a3b7-1d72d3152714-tls-certs\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.469726 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.469624 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-dshm\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.469726 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.469648 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-model-cache\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.469726 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.469693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-home\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.570781 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.570742 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-tmp-dir\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.570781 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.570784 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f80b73-c836-46b4-a3b7-1d72d3152714-tls-certs\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.571019 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.570833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-dshm\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.571019 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.570982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-model-cache\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.571101 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.571023 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-home\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.571101 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.571093 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjdxx\" (UniqueName: \"kubernetes.io/projected/69f80b73-c836-46b4-a3b7-1d72d3152714-kube-api-access-cjdxx\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.571236 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.571217 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-tmp-dir\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.571362 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.571336 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-model-cache\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.571429 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.571390 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-home\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.573217 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.573195 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-dshm\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.573461 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.573443 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f80b73-c836-46b4-a3b7-1d72d3152714-tls-certs\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.587062 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.587038 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjdxx\" (UniqueName: \"kubernetes.io/projected/69f80b73-c836-46b4-a3b7-1d72d3152714-kube-api-access-cjdxx\") pod \"auth-test-model-065403-kserve-7748564b99-2l2gz\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.722977 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.722856 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:05.869935 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:05.869910 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz"] Jul 21 20:17:05.870639 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:17:05.870616 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69f80b73_c836_46b4_a3b7_1d72d3152714.slice/crio-db4a0a3b3f8760e48c2ff297e6787a78a42934ca33be7b0f9ba2ebb94b98ba3c WatchSource:0}: Error finding container db4a0a3b3f8760e48c2ff297e6787a78a42934ca33be7b0f9ba2ebb94b98ba3c: Status 404 returned error can't find the container with id db4a0a3b3f8760e48c2ff297e6787a78a42934ca33be7b0f9ba2ebb94b98ba3c Jul 21 20:17:06.114644 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:06.114556 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" event={"ID":"69f80b73-c836-46b4-a3b7-1d72d3152714","Type":"ContainerStarted","Data":"bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c"} Jul 21 20:17:06.114644 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:06.114592 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" event={"ID":"69f80b73-c836-46b4-a3b7-1d72d3152714","Type":"ContainerStarted","Data":"db4a0a3b3f8760e48c2ff297e6787a78a42934ca33be7b0f9ba2ebb94b98ba3c"} Jul 21 20:17:06.136228 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:06.136182 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" podStartSLOduration=1.136166448 podStartE2EDuration="1.136166448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:17:06.135205642 +0000 UTC m=+2389.319647347" watchObservedRunningTime="2026-07-21 20:17:06.136166448 +0000 UTC m=+2389.320608129" Jul 21 20:17:08.821463 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:08.821420 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:17:15.723616 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:15.723577 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:15.723616 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:15.723629 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:15.736336 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:15.736310 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:16.188438 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:16.188409 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:17:17.391144 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:17.391115 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:17:17.395181 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:17.395148 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:17:36.413159 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:36.413124 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:17:59.271402 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:17:59.271324 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:18:12.203752 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:18:12.203711 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:18:29.512774 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:18:29.512722 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:18:43.109084 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:18:43.109031 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:19:00.622497 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:19:00.618700 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:19:17.915564 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:19:17.915486 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:19:36.827459 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:19:36.827426 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:19:49.115291 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:19:49.115256 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz"] Jul 21 20:19:55.705225 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:19:55.705192 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:20:03.022223 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.022184 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4"] Jul 21 20:20:03.025720 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.025702 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.028910 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.028890 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-065403\"/\"sub-test-model-065403-kserve-self-signed-certs\"" Jul 21 20:20:03.035257 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.035233 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4"] Jul 21 20:20:03.189856 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.189804 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-model-cache\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.190106 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.189963 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec81477-faaa-451d-b340-b02ae04b8f5e-tls-certs\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.190106 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.190014 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-tmp-dir\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.190106 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.190051 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-dshm\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.190106 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.190084 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlw9t\" (UniqueName: \"kubernetes.io/projected/dec81477-faaa-451d-b340-b02ae04b8f5e-kube-api-access-rlw9t\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.190304 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.190135 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-home\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.291316 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.291225 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-dshm\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.291316 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.291274 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlw9t\" (UniqueName: \"kubernetes.io/projected/dec81477-faaa-451d-b340-b02ae04b8f5e-kube-api-access-rlw9t\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.291316 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.291304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-home\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.291570 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.291340 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-model-cache\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.291570 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.291404 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec81477-faaa-451d-b340-b02ae04b8f5e-tls-certs\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.291570 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.291424 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-tmp-dir\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.291772 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.291747 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-home\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.291843 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.291781 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-tmp-dir\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.291843 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.291832 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-model-cache\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.293652 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.293631 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-dshm\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.293837 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.293822 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec81477-faaa-451d-b340-b02ae04b8f5e-tls-certs\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.300359 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.300336 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlw9t\" (UniqueName: \"kubernetes.io/projected/dec81477-faaa-451d-b340-b02ae04b8f5e-kube-api-access-rlw9t\") pod \"sub-test-model-065403-kserve-844dfd94dc-9qqk4\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.336399 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.336365 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:03.479628 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:03.479597 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4"] Jul 21 20:20:03.481382 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:20:03.481337 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddec81477_faaa_451d_b340_b02ae04b8f5e.slice/crio-b3d5ae051879d378d8c648bd5fc21fc16f569eeab93d48b716c72ea9fb25fe08 WatchSource:0}: Error finding container b3d5ae051879d378d8c648bd5fc21fc16f569eeab93d48b716c72ea9fb25fe08: Status 404 returned error can't find the container with id b3d5ae051879d378d8c648bd5fc21fc16f569eeab93d48b716c72ea9fb25fe08 Jul 21 20:20:04.080120 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.080082 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" event={"ID":"dec81477-faaa-451d-b340-b02ae04b8f5e","Type":"ContainerStarted","Data":"ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0"} Jul 21 20:20:04.080509 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.080129 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" event={"ID":"dec81477-faaa-451d-b340-b02ae04b8f5e","Type":"ContainerStarted","Data":"b3d5ae051879d378d8c648bd5fc21fc16f569eeab93d48b716c72ea9fb25fe08"} Jul 21 20:20:04.101923 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.101846 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" podStartSLOduration=1.1018299950000001 podStartE2EDuration="1.101829995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:20:04.100632812 +0000 UTC m=+2567.285074494" watchObservedRunningTime="2026-07-21 20:20:04.101829995 +0000 UTC m=+2567.286271675" Jul 21 20:20:04.135082 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.135039 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" podUID="69f80b73-c836-46b4-a3b7-1d72d3152714" containerName="main" containerID="cri-o://bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c" gracePeriod=45 Jul 21 20:20:04.400926 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.400899 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:20:04.503510 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.503482 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-tmp-dir\") pod \"69f80b73-c836-46b4-a3b7-1d72d3152714\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " Jul 21 20:20:04.503700 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.503530 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-model-cache\") pod \"69f80b73-c836-46b4-a3b7-1d72d3152714\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " Jul 21 20:20:04.503700 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.503553 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjdxx\" (UniqueName: \"kubernetes.io/projected/69f80b73-c836-46b4-a3b7-1d72d3152714-kube-api-access-cjdxx\") pod \"69f80b73-c836-46b4-a3b7-1d72d3152714\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " Jul 21 20:20:04.503700 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.503613 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f80b73-c836-46b4-a3b7-1d72d3152714-tls-certs\") pod \"69f80b73-c836-46b4-a3b7-1d72d3152714\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " Jul 21 20:20:04.503700 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.503640 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-home\") pod \"69f80b73-c836-46b4-a3b7-1d72d3152714\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " Jul 21 20:20:04.503700 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.503685 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-dshm\") pod \"69f80b73-c836-46b4-a3b7-1d72d3152714\" (UID: \"69f80b73-c836-46b4-a3b7-1d72d3152714\") " Jul 21 20:20:04.503996 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.503766 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69f80b73-c836-46b4-a3b7-1d72d3152714" (UID: "69f80b73-c836-46b4-a3b7-1d72d3152714"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:20:04.503996 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.503812 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-model-cache" (OuterVolumeSpecName: "model-cache") pod "69f80b73-c836-46b4-a3b7-1d72d3152714" (UID: "69f80b73-c836-46b4-a3b7-1d72d3152714"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:20:04.504089 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.504015 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-home" (OuterVolumeSpecName: "home") pod "69f80b73-c836-46b4-a3b7-1d72d3152714" (UID: "69f80b73-c836-46b4-a3b7-1d72d3152714"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:20:04.504089 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.504017 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-tmp-dir\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:20:04.504089 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.504053 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-model-cache\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:20:04.506104 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.506076 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f80b73-c836-46b4-a3b7-1d72d3152714-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69f80b73-c836-46b4-a3b7-1d72d3152714" (UID: "69f80b73-c836-46b4-a3b7-1d72d3152714"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:20:04.506215 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.506164 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f80b73-c836-46b4-a3b7-1d72d3152714-kube-api-access-cjdxx" (OuterVolumeSpecName: "kube-api-access-cjdxx") pod "69f80b73-c836-46b4-a3b7-1d72d3152714" (UID: "69f80b73-c836-46b4-a3b7-1d72d3152714"). InnerVolumeSpecName "kube-api-access-cjdxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:20:04.506215 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.506193 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-dshm" (OuterVolumeSpecName: "dshm") pod "69f80b73-c836-46b4-a3b7-1d72d3152714" (UID: "69f80b73-c836-46b4-a3b7-1d72d3152714"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:20:04.605372 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.605338 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f80b73-c836-46b4-a3b7-1d72d3152714-tls-certs\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:20:04.605372 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.605365 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-home\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:20:04.605372 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.605373 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f80b73-c836-46b4-a3b7-1d72d3152714-dshm\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:20:04.605372 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:04.605383 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjdxx\" (UniqueName: \"kubernetes.io/projected/69f80b73-c836-46b4-a3b7-1d72d3152714-kube-api-access-cjdxx\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:20:05.086831 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:05.086799 2557 generic.go:356] "Generic (PLEG): container finished" podID="69f80b73-c836-46b4-a3b7-1d72d3152714" containerID="bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c" exitCode=0 Jul 21 20:20:05.087330 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:05.086848 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" Jul 21 20:20:05.087330 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:05.086901 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" event={"ID":"69f80b73-c836-46b4-a3b7-1d72d3152714","Type":"ContainerDied","Data":"bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c"} Jul 21 20:20:05.087330 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:05.086937 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz" event={"ID":"69f80b73-c836-46b4-a3b7-1d72d3152714","Type":"ContainerDied","Data":"db4a0a3b3f8760e48c2ff297e6787a78a42934ca33be7b0f9ba2ebb94b98ba3c"} Jul 21 20:20:05.087330 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:05.086954 2557 scope.go:117] "RemoveContainer" containerID="bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c" Jul 21 20:20:05.103664 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:05.103645 2557 scope.go:117] "RemoveContainer" containerID="bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c" Jul 21 20:20:05.103977 ip-10-0-134-158 kubenswrapper[2557]: E0721 20:20:05.103951 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c\": container with ID starting with bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c not found: ID does not exist" containerID="bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c" Jul 21 20:20:05.104075 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:05.103983 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c"} err="failed to get container status \"bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c\": rpc error: code = NotFound desc = could not find container \"bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c\": container with ID starting with bf8e487db397d01b41fbaf81507f686ab1138c47d86fa4f821550e24ee9f916c not found: ID does not exist" Jul 21 20:20:05.119938 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:05.119905 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz"] Jul 21 20:20:05.122068 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:05.122041 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-065403/auth-test-model-065403-kserve-7748564b99-2l2gz"] Jul 21 20:20:06.104169 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:06.104137 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:20:06.505917 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:06.505819 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f80b73-c836-46b4-a3b7-1d72d3152714" path="/var/lib/kubelet/pods/69f80b73-c836-46b4-a3b7-1d72d3152714/volumes" Jul 21 20:20:13.337508 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:13.337469 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:13.338042 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:13.337609 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:13.350950 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:13.350923 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:14.158686 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:14.158651 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:20:27.114231 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:27.114192 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:20:50.398175 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:20:50.398096 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:21:08.016585 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:08.016549 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:21:15.145312 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:15.145274 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4"] Jul 21 20:21:26.898760 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:26.898716 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:21:30.165250 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.165208 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" podUID="dec81477-faaa-451d-b340-b02ae04b8f5e" containerName="main" containerID="cri-o://ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0" gracePeriod=45 Jul 21 20:21:30.423989 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.423925 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:21:30.515031 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515005 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-home\") pod \"dec81477-faaa-451d-b340-b02ae04b8f5e\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " Jul 21 20:21:30.515266 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515075 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-model-cache\") pod \"dec81477-faaa-451d-b340-b02ae04b8f5e\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " Jul 21 20:21:30.515266 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515153 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-dshm\") pod \"dec81477-faaa-451d-b340-b02ae04b8f5e\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " Jul 21 20:21:30.515266 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515204 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-tmp-dir\") pod \"dec81477-faaa-451d-b340-b02ae04b8f5e\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " Jul 21 20:21:30.515266 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515263 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlw9t\" (UniqueName: \"kubernetes.io/projected/dec81477-faaa-451d-b340-b02ae04b8f5e-kube-api-access-rlw9t\") pod \"dec81477-faaa-451d-b340-b02ae04b8f5e\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " Jul 21 20:21:30.515481 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515258 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-home" (OuterVolumeSpecName: "home") pod "dec81477-faaa-451d-b340-b02ae04b8f5e" (UID: "dec81477-faaa-451d-b340-b02ae04b8f5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:21:30.515481 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515296 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec81477-faaa-451d-b340-b02ae04b8f5e-tls-certs\") pod \"dec81477-faaa-451d-b340-b02ae04b8f5e\" (UID: \"dec81477-faaa-451d-b340-b02ae04b8f5e\") " Jul 21 20:21:30.515481 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515321 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "dec81477-faaa-451d-b340-b02ae04b8f5e" (UID: "dec81477-faaa-451d-b340-b02ae04b8f5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:21:30.515628 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515492 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dec81477-faaa-451d-b340-b02ae04b8f5e" (UID: "dec81477-faaa-451d-b340-b02ae04b8f5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:21:30.515628 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515588 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-home\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:21:30.515628 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515605 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-model-cache\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:21:30.515628 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.515619 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-tmp-dir\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:21:30.517368 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.517345 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-dshm" (OuterVolumeSpecName: "dshm") pod "dec81477-faaa-451d-b340-b02ae04b8f5e" (UID: "dec81477-faaa-451d-b340-b02ae04b8f5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:21:30.517368 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.517351 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec81477-faaa-451d-b340-b02ae04b8f5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dec81477-faaa-451d-b340-b02ae04b8f5e" (UID: "dec81477-faaa-451d-b340-b02ae04b8f5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:21:30.517501 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.517377 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec81477-faaa-451d-b340-b02ae04b8f5e-kube-api-access-rlw9t" (OuterVolumeSpecName: "kube-api-access-rlw9t") pod "dec81477-faaa-451d-b340-b02ae04b8f5e" (UID: "dec81477-faaa-451d-b340-b02ae04b8f5e"). InnerVolumeSpecName "kube-api-access-rlw9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:21:30.562038 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.562001 2557 generic.go:356] "Generic (PLEG): container finished" podID="dec81477-faaa-451d-b340-b02ae04b8f5e" containerID="ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0" exitCode=0 Jul 21 20:21:30.562223 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.562049 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" event={"ID":"dec81477-faaa-451d-b340-b02ae04b8f5e","Type":"ContainerDied","Data":"ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0"} Jul 21 20:21:30.562223 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.562091 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" event={"ID":"dec81477-faaa-451d-b340-b02ae04b8f5e","Type":"ContainerDied","Data":"b3d5ae051879d378d8c648bd5fc21fc16f569eeab93d48b716c72ea9fb25fe08"} Jul 21 20:21:30.562223 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.562112 2557 scope.go:117] "RemoveContainer" containerID="ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0" Jul 21 20:21:30.562223 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.562056 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4" Jul 21 20:21:30.571967 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.571949 2557 scope.go:117] "RemoveContainer" containerID="ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0" Jul 21 20:21:30.572262 ip-10-0-134-158 kubenswrapper[2557]: E0721 20:21:30.572243 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0\": container with ID starting with ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0 not found: ID does not exist" containerID="ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0" Jul 21 20:21:30.572332 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.572273 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0"} err="failed to get container status \"ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0\": rpc error: code = NotFound desc = could not find container \"ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0\": container with ID starting with ba83004a646b38b1efd67efe1a16c2d10ac02a190b9d4244397f5768f8a7e9a0 not found: ID does not exist" Jul 21 20:21:30.585663 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.585627 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4"] Jul 21 20:21:30.587778 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.587759 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-065403/sub-test-model-065403-kserve-844dfd94dc-9qqk4"] Jul 21 20:21:30.616644 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.616615 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlw9t\" (UniqueName: \"kubernetes.io/projected/dec81477-faaa-451d-b340-b02ae04b8f5e-kube-api-access-rlw9t\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:21:30.616644 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.616641 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec81477-faaa-451d-b340-b02ae04b8f5e-tls-certs\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:21:30.616644 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:30.616652 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec81477-faaa-451d-b340-b02ae04b8f5e-dshm\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:21:32.505556 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:32.505520 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dec81477-faaa-451d-b340-b02ae04b8f5e" path="/var/lib/kubelet/pods/dec81477-faaa-451d-b340-b02ae04b8f5e/volumes" Jul 21 20:21:37.130485 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.130449 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776"] Jul 21 20:21:37.131033 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.131015 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec81477-faaa-451d-b340-b02ae04b8f5e" containerName="main" Jul 21 20:21:37.131127 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.131035 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec81477-faaa-451d-b340-b02ae04b8f5e" containerName="main" Jul 21 20:21:37.131127 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.131076 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f80b73-c836-46b4-a3b7-1d72d3152714" containerName="main" Jul 21 20:21:37.131127 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.131085 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f80b73-c836-46b4-a3b7-1d72d3152714" containerName="main" Jul 21 20:21:37.131281 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.131189 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dec81477-faaa-451d-b340-b02ae04b8f5e" containerName="main" Jul 21 20:21:37.131281 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.131204 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f80b73-c836-46b4-a3b7-1d72d3152714" containerName="main" Jul 21 20:21:37.135675 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.135653 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.140184 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.140162 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-065403\"/\"default-dockercfg-2qvt5\"" Jul 21 20:21:37.140184 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.140179 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-065403\"/\"openshift-service-ca.crt\"" Jul 21 20:21:37.140338 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.140187 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-065403\"/\"rate-test-model-065403-kserve-self-signed-certs\"" Jul 21 20:21:37.140338 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.140206 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-065403\"/\"kube-root-ca.crt\"" Jul 21 20:21:37.148241 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.148218 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776"] Jul 21 20:21:37.278957 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.278917 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tls-certs\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.279178 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.279020 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-home\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.279178 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.279046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4qb2\" (UniqueName: \"kubernetes.io/projected/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-kube-api-access-g4qb2\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.279178 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.279073 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-dshm\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.279178 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.279159 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tmp-dir\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.279349 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.279183 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-model-cache\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.380051 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.380009 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-home\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.380227 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.380059 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qb2\" (UniqueName: \"kubernetes.io/projected/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-kube-api-access-g4qb2\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.380227 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.380095 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-dshm\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.380227 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.380136 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tmp-dir\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.380227 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.380160 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-model-cache\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.380227 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.380213 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tls-certs\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.380590 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.380532 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-home\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.380590 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.380574 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tmp-dir\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.380710 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.380665 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-model-cache\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.382672 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.382650 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-dshm\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.382787 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.382766 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tls-certs\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.390571 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.390544 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qb2\" (UniqueName: \"kubernetes.io/projected/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-kube-api-access-g4qb2\") pod \"rate-test-model-065403-kserve-6c886c7899-ww776\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.446370 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.446334 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:37.595066 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:37.595035 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776"] Jul 21 20:21:37.597106 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:21:37.597074 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb9f3c55_7da2_4ce7_9689_b0bfd901e2d9.slice/crio-e97e454f752d00e01f4c939cbb22d0601f60b2df9d05dc91ffabbed12fbb8b2d WatchSource:0}: Error finding container e97e454f752d00e01f4c939cbb22d0601f60b2df9d05dc91ffabbed12fbb8b2d: Status 404 returned error can't find the container with id e97e454f752d00e01f4c939cbb22d0601f60b2df9d05dc91ffabbed12fbb8b2d Jul 21 20:21:38.607185 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:38.607143 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" event={"ID":"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9","Type":"ContainerStarted","Data":"c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023"} Jul 21 20:21:38.607185 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:38.607190 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" event={"ID":"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9","Type":"ContainerStarted","Data":"e97e454f752d00e01f4c939cbb22d0601f60b2df9d05dc91ffabbed12fbb8b2d"} Jul 21 20:21:38.640943 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:38.640869 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" podStartSLOduration=1.640853957 podStartE2EDuration="1.640853957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:21:38.639337815 +0000 UTC m=+2661.823779517" watchObservedRunningTime="2026-07-21 20:21:38.640853957 +0000 UTC m=+2661.825295639" Jul 21 20:21:40.426256 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:40.426220 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:21:47.446771 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:47.446734 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:47.447208 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:47.446903 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:47.460147 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:47.460121 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:21:47.666859 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:21:47.666822 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:22:17.425431 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:22:17.425346 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:22:17.428791 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:22:17.428769 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:22:25.605336 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:22:25.605303 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:22:40.196405 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:22:40.196368 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:22:59.707524 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:22:59.707493 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:23:05.832461 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:05.832428 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776"] Jul 21 20:23:07.852520 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:07.852482 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7441-140614109842480"] Jul 21 20:23:07.856108 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:07.856091 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109842480" Jul 21 20:23:07.858757 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:07.858734 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z9cgq\"" Jul 21 20:23:07.862564 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:07.862540 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109842480"] Jul 21 20:23:08.031176 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:08.031139 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49kv9\" (UniqueName: \"kubernetes.io/projected/c3e5bb7e-e90a-4bfc-bcb7-3925714337d0-kube-api-access-49kv9\") pod \"test-curl-7441-140614109842480\" (UID: \"c3e5bb7e-e90a-4bfc-bcb7-3925714337d0\") " pod="opendatahub/test-curl-7441-140614109842480" Jul 21 20:23:08.132521 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:08.132432 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49kv9\" (UniqueName: \"kubernetes.io/projected/c3e5bb7e-e90a-4bfc-bcb7-3925714337d0-kube-api-access-49kv9\") pod \"test-curl-7441-140614109842480\" (UID: \"c3e5bb7e-e90a-4bfc-bcb7-3925714337d0\") " pod="opendatahub/test-curl-7441-140614109842480" Jul 21 20:23:08.142932 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:08.142905 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49kv9\" (UniqueName: \"kubernetes.io/projected/c3e5bb7e-e90a-4bfc-bcb7-3925714337d0-kube-api-access-49kv9\") pod \"test-curl-7441-140614109842480\" (UID: \"c3e5bb7e-e90a-4bfc-bcb7-3925714337d0\") " pod="opendatahub/test-curl-7441-140614109842480" Jul 21 20:23:08.167682 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:08.167654 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109842480" Jul 21 20:23:08.311759 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:08.311621 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109842480"] Jul 21 20:23:08.314674 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:23:08.314649 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3e5bb7e_e90a_4bfc_bcb7_3925714337d0.slice/crio-6927850f14f497dcb5a7d384b334f635fecd9600112487d04b0634d5ff4f647b WatchSource:0}: Error finding container 6927850f14f497dcb5a7d384b334f635fecd9600112487d04b0634d5ff4f647b: Status 404 returned error can't find the container with id 6927850f14f497dcb5a7d384b334f635fecd9600112487d04b0634d5ff4f647b Jul 21 20:23:08.316135 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:08.316118 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 20:23:09.108771 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:09.108747 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109842480" event={"ID":"c3e5bb7e-e90a-4bfc-bcb7-3925714337d0","Type":"ContainerStarted","Data":"6927850f14f497dcb5a7d384b334f635fecd9600112487d04b0634d5ff4f647b"} Jul 21 20:23:10.115812 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:10.115766 2557 generic.go:356] "Generic (PLEG): container finished" podID="c3e5bb7e-e90a-4bfc-bcb7-3925714337d0" containerID="0f1add684f85d1ce85b1361a66b738d65f2da1bc9238b3657f9ff2423f9501a3" exitCode=0 Jul 21 20:23:10.116205 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:10.115834 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109842480" event={"ID":"c3e5bb7e-e90a-4bfc-bcb7-3925714337d0","Type":"ContainerDied","Data":"0f1add684f85d1ce85b1361a66b738d65f2da1bc9238b3657f9ff2423f9501a3"} Jul 21 20:23:11.264288 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.264265 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109842480" Jul 21 20:23:11.303560 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.303518 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7441-140614109842480_c3e5bb7e-e90a-4bfc-bcb7-3925714337d0/test-curl-7441-140614109842480/0.log" Jul 21 20:23:11.332847 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.332812 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7441-140614109842480"] Jul 21 20:23:11.337012 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.336986 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7441-140614109842480"] Jul 21 20:23:11.358122 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.358089 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49kv9\" (UniqueName: \"kubernetes.io/projected/c3e5bb7e-e90a-4bfc-bcb7-3925714337d0-kube-api-access-49kv9\") pod \"c3e5bb7e-e90a-4bfc-bcb7-3925714337d0\" (UID: \"c3e5bb7e-e90a-4bfc-bcb7-3925714337d0\") " Jul 21 20:23:11.360288 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.360260 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e5bb7e-e90a-4bfc-bcb7-3925714337d0-kube-api-access-49kv9" (OuterVolumeSpecName: "kube-api-access-49kv9") pod "c3e5bb7e-e90a-4bfc-bcb7-3925714337d0" (UID: "c3e5bb7e-e90a-4bfc-bcb7-3925714337d0"). InnerVolumeSpecName "kube-api-access-49kv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:23:11.459372 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.459340 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49kv9\" (UniqueName: \"kubernetes.io/projected/c3e5bb7e-e90a-4bfc-bcb7-3925714337d0-kube-api-access-49kv9\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:11.472510 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.472477 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7441-140614109807664"] Jul 21 20:23:11.472928 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.472913 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e5bb7e-e90a-4bfc-bcb7-3925714337d0" containerName="test-curl-7441-140614109842480" Jul 21 20:23:11.472985 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.472929 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e5bb7e-e90a-4bfc-bcb7-3925714337d0" containerName="test-curl-7441-140614109842480" Jul 21 20:23:11.473029 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.472993 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e5bb7e-e90a-4bfc-bcb7-3925714337d0" containerName="test-curl-7441-140614109842480" Jul 21 20:23:11.476141 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.476125 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109807664" Jul 21 20:23:11.482162 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.482135 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109807664"] Jul 21 20:23:11.661391 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.661357 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrkd2\" (UniqueName: \"kubernetes.io/projected/eb9716e9-cb1e-40d3-b70e-186805b06f4e-kube-api-access-jrkd2\") pod \"test-curl-7441-140614109807664\" (UID: \"eb9716e9-cb1e-40d3-b70e-186805b06f4e\") " pod="opendatahub/test-curl-7441-140614109807664" Jul 21 20:23:11.762482 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.762383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkd2\" (UniqueName: \"kubernetes.io/projected/eb9716e9-cb1e-40d3-b70e-186805b06f4e-kube-api-access-jrkd2\") pod \"test-curl-7441-140614109807664\" (UID: \"eb9716e9-cb1e-40d3-b70e-186805b06f4e\") " pod="opendatahub/test-curl-7441-140614109807664" Jul 21 20:23:11.771282 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.771247 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkd2\" (UniqueName: \"kubernetes.io/projected/eb9716e9-cb1e-40d3-b70e-186805b06f4e-kube-api-access-jrkd2\") pod \"test-curl-7441-140614109807664\" (UID: \"eb9716e9-cb1e-40d3-b70e-186805b06f4e\") " pod="opendatahub/test-curl-7441-140614109807664" Jul 21 20:23:11.787206 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.787182 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109807664" Jul 21 20:23:11.925063 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:11.925035 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109807664"] Jul 21 20:23:11.926172 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:23:11.926150 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb9716e9_cb1e_40d3_b70e_186805b06f4e.slice/crio-baf08f25d5eebafa89cedb3e325e50238627947767f78216c60257e758c2132a WatchSource:0}: Error finding container baf08f25d5eebafa89cedb3e325e50238627947767f78216c60257e758c2132a: Status 404 returned error can't find the container with id baf08f25d5eebafa89cedb3e325e50238627947767f78216c60257e758c2132a Jul 21 20:23:12.128996 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:12.128914 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109842480" Jul 21 20:23:12.128996 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:12.128937 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6927850f14f497dcb5a7d384b334f635fecd9600112487d04b0634d5ff4f647b" Jul 21 20:23:12.130279 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:12.130253 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109807664" event={"ID":"eb9716e9-cb1e-40d3-b70e-186805b06f4e","Type":"ContainerStarted","Data":"baf08f25d5eebafa89cedb3e325e50238627947767f78216c60257e758c2132a"} Jul 21 20:23:12.140002 ip-10-0-134-158 kubenswrapper[2557]: E0721 20:23:12.139977 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7441-140614109842480\" is forbidden: User \"system:node:ip-10-0-134-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-158.ec2.internal' and this object" podUID="c3e5bb7e-e90a-4bfc-bcb7-3925714337d0" pod="opendatahub/test-curl-7441-140614109842480" Jul 21 20:23:12.506095 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:12.506065 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e5bb7e-e90a-4bfc-bcb7-3925714337d0" path="/var/lib/kubelet/pods/c3e5bb7e-e90a-4bfc-bcb7-3925714337d0/volumes" Jul 21 20:23:13.137579 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:13.137542 2557 generic.go:356] "Generic (PLEG): container finished" podID="eb9716e9-cb1e-40d3-b70e-186805b06f4e" containerID="9608ac845ecc8b429ef231ce372eae015b5695dbf99c7a550e59448f1583cc1e" exitCode=0 Jul 21 20:23:13.137784 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:13.137623 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109807664" event={"ID":"eb9716e9-cb1e-40d3-b70e-186805b06f4e","Type":"ContainerDied","Data":"9608ac845ecc8b429ef231ce372eae015b5695dbf99c7a550e59448f1583cc1e"} Jul 21 20:23:14.284599 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.284573 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109807664" Jul 21 20:23:14.307053 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.307026 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7441-140614109807664_eb9716e9-cb1e-40d3-b70e-186805b06f4e/test-curl-7441-140614109807664/0.log" Jul 21 20:23:14.332292 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.332261 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7441-140614109807664"] Jul 21 20:23:14.338041 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.338014 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7441-140614109807664"] Jul 21 20:23:14.384182 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.384149 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrkd2\" (UniqueName: \"kubernetes.io/projected/eb9716e9-cb1e-40d3-b70e-186805b06f4e-kube-api-access-jrkd2\") pod \"eb9716e9-cb1e-40d3-b70e-186805b06f4e\" (UID: \"eb9716e9-cb1e-40d3-b70e-186805b06f4e\") " Jul 21 20:23:14.386243 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.386221 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9716e9-cb1e-40d3-b70e-186805b06f4e-kube-api-access-jrkd2" (OuterVolumeSpecName: "kube-api-access-jrkd2") pod "eb9716e9-cb1e-40d3-b70e-186805b06f4e" (UID: "eb9716e9-cb1e-40d3-b70e-186805b06f4e"). InnerVolumeSpecName "kube-api-access-jrkd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:23:14.485797 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.485712 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrkd2\" (UniqueName: \"kubernetes.io/projected/eb9716e9-cb1e-40d3-b70e-186805b06f4e-kube-api-access-jrkd2\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:14.505639 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.505607 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9716e9-cb1e-40d3-b70e-186805b06f4e" path="/var/lib/kubelet/pods/eb9716e9-cb1e-40d3-b70e-186805b06f4e/volumes" Jul 21 20:23:14.541212 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.541181 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:14.541574 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.541561 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9716e9-cb1e-40d3-b70e-186805b06f4e" containerName="test-curl-7441-140614109807664" Jul 21 20:23:14.541623 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.541576 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9716e9-cb1e-40d3-b70e-186805b06f4e" containerName="test-curl-7441-140614109807664" Jul 21 20:23:14.541664 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.541654 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9716e9-cb1e-40d3-b70e-186805b06f4e" containerName="test-curl-7441-140614109807664" Jul 21 20:23:14.544617 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.544598 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:14.549186 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.549158 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:14.587039 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.587011 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vghwf\" (UniqueName: \"kubernetes.io/projected/63bc2d40-3e37-4db8-862a-f5a07701e0e7-kube-api-access-vghwf\") pod \"test-curl-7441-140614109791264\" (UID: \"63bc2d40-3e37-4db8-862a-f5a07701e0e7\") " pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:14.687759 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.687722 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vghwf\" (UniqueName: \"kubernetes.io/projected/63bc2d40-3e37-4db8-862a-f5a07701e0e7-kube-api-access-vghwf\") pod \"test-curl-7441-140614109791264\" (UID: \"63bc2d40-3e37-4db8-862a-f5a07701e0e7\") " pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:14.697084 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.697053 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vghwf\" (UniqueName: \"kubernetes.io/projected/63bc2d40-3e37-4db8-862a-f5a07701e0e7-kube-api-access-vghwf\") pod \"test-curl-7441-140614109791264\" (UID: \"63bc2d40-3e37-4db8-862a-f5a07701e0e7\") " pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:14.856598 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.856510 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:14.996890 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:14.996846 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:14.998422 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:23:14.998401 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63bc2d40_3e37_4db8_862a_f5a07701e0e7.slice/crio-48b803f61e3fc999119d5952caba7ca3f1efb7d9581f34df020d423f6ae87c87 WatchSource:0}: Error finding container 48b803f61e3fc999119d5952caba7ca3f1efb7d9581f34df020d423f6ae87c87: Status 404 returned error can't find the container with id 48b803f61e3fc999119d5952caba7ca3f1efb7d9581f34df020d423f6ae87c87 Jul 21 20:23:15.149670 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:15.149631 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109791264" event={"ID":"63bc2d40-3e37-4db8-862a-f5a07701e0e7","Type":"ContainerStarted","Data":"48b803f61e3fc999119d5952caba7ca3f1efb7d9581f34df020d423f6ae87c87"} Jul 21 20:23:15.150971 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:15.150945 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109807664" Jul 21 20:23:15.150971 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:15.150962 2557 scope.go:117] "RemoveContainer" containerID="9608ac845ecc8b429ef231ce372eae015b5695dbf99c7a550e59448f1583cc1e" Jul 21 20:23:16.159001 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:16.158964 2557 generic.go:356] "Generic (PLEG): container finished" podID="63bc2d40-3e37-4db8-862a-f5a07701e0e7" containerID="1a06ecc1eb07458c7bcfeaa4a192fb59950d36019850eee1469181066bca2846" exitCode=0 Jul 21 20:23:16.159394 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:16.159005 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109791264" event={"ID":"63bc2d40-3e37-4db8-862a-f5a07701e0e7","Type":"ContainerDied","Data":"1a06ecc1eb07458c7bcfeaa4a192fb59950d36019850eee1469181066bca2846"} Jul 21 20:23:17.310373 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.310346 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:17.334089 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.334061 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7441-140614109791264_63bc2d40-3e37-4db8-862a-f5a07701e0e7/test-curl-7441-140614109791264/0.log" Jul 21 20:23:17.362998 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.362967 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:17.365111 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.365090 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:17.415535 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.415501 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vghwf\" (UniqueName: \"kubernetes.io/projected/63bc2d40-3e37-4db8-862a-f5a07701e0e7-kube-api-access-vghwf\") pod \"63bc2d40-3e37-4db8-862a-f5a07701e0e7\" (UID: \"63bc2d40-3e37-4db8-862a-f5a07701e0e7\") " Jul 21 20:23:17.417621 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.417548 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bc2d40-3e37-4db8-862a-f5a07701e0e7-kube-api-access-vghwf" (OuterVolumeSpecName: "kube-api-access-vghwf") pod "63bc2d40-3e37-4db8-862a-f5a07701e0e7" (UID: "63bc2d40-3e37-4db8-862a-f5a07701e0e7"). InnerVolumeSpecName "kube-api-access-vghwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:23:17.498351 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.498319 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:17.498710 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.498698 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63bc2d40-3e37-4db8-862a-f5a07701e0e7" containerName="test-curl-7441-140614109791264" Jul 21 20:23:17.498751 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.498711 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="63bc2d40-3e37-4db8-862a-f5a07701e0e7" containerName="test-curl-7441-140614109791264" Jul 21 20:23:17.498788 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.498777 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="63bc2d40-3e37-4db8-862a-f5a07701e0e7" containerName="test-curl-7441-140614109791264" Jul 21 20:23:17.501713 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.501696 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:17.510088 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.510060 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:17.517257 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.517234 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dbt\" (UniqueName: \"kubernetes.io/projected/b4fe4647-251d-45e2-9a17-283d644482b7-kube-api-access-j7dbt\") pod \"test-curl-7441-140614109791264\" (UID: \"b4fe4647-251d-45e2-9a17-283d644482b7\") " pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:17.517393 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.517304 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vghwf\" (UniqueName: \"kubernetes.io/projected/63bc2d40-3e37-4db8-862a-f5a07701e0e7-kube-api-access-vghwf\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:17.618740 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.618705 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dbt\" (UniqueName: \"kubernetes.io/projected/b4fe4647-251d-45e2-9a17-283d644482b7-kube-api-access-j7dbt\") pod \"test-curl-7441-140614109791264\" (UID: \"b4fe4647-251d-45e2-9a17-283d644482b7\") " pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:17.627921 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.627904 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dbt\" (UniqueName: \"kubernetes.io/projected/b4fe4647-251d-45e2-9a17-283d644482b7-kube-api-access-j7dbt\") pod \"test-curl-7441-140614109791264\" (UID: \"b4fe4647-251d-45e2-9a17-283d644482b7\") " pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:17.811505 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.811422 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:17.952426 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:17.952395 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:17.953218 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:23:17.953188 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4fe4647_251d_45e2_9a17_283d644482b7.slice/crio-e4833562aafb5b5efcc3aa97a62eea300592fc855411ecf06fa161929ff5e54c WatchSource:0}: Error finding container e4833562aafb5b5efcc3aa97a62eea300592fc855411ecf06fa161929ff5e54c: Status 404 returned error can't find the container with id e4833562aafb5b5efcc3aa97a62eea300592fc855411ecf06fa161929ff5e54c Jul 21 20:23:18.174005 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:18.173973 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:18.174005 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:18.174006 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48b803f61e3fc999119d5952caba7ca3f1efb7d9581f34df020d423f6ae87c87" Jul 21 20:23:18.175313 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:18.175288 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109791264" event={"ID":"b4fe4647-251d-45e2-9a17-283d644482b7","Type":"ContainerStarted","Data":"e4833562aafb5b5efcc3aa97a62eea300592fc855411ecf06fa161929ff5e54c"} Jul 21 20:23:18.185813 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:18.185785 2557 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7441-140614109791264" oldPodUID="63bc2d40-3e37-4db8-862a-f5a07701e0e7" podUID="b4fe4647-251d-45e2-9a17-283d644482b7" Jul 21 20:23:18.505329 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:18.505254 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63bc2d40-3e37-4db8-862a-f5a07701e0e7" path="/var/lib/kubelet/pods/63bc2d40-3e37-4db8-862a-f5a07701e0e7/volumes" Jul 21 20:23:19.003101 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:19.003066 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:23:19.182745 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:19.182706 2557 generic.go:356] "Generic (PLEG): container finished" podID="b4fe4647-251d-45e2-9a17-283d644482b7" containerID="b2e5348a1e8eb7a85d6a2d3dc2fcf79c1d3e7cb6970656df95d0bba6a540a646" exitCode=0 Jul 21 20:23:19.182906 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:19.182757 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109791264" event={"ID":"b4fe4647-251d-45e2-9a17-283d644482b7","Type":"ContainerDied","Data":"b2e5348a1e8eb7a85d6a2d3dc2fcf79c1d3e7cb6970656df95d0bba6a540a646"} Jul 21 20:23:20.330544 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.330520 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:20.353183 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.353154 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7441-140614109791264_b4fe4647-251d-45e2-9a17-283d644482b7/test-curl-7441-140614109791264/0.log" Jul 21 20:23:20.377621 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.377584 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:20.383997 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.383973 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7441-140614109791264"] Jul 21 20:23:20.445342 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.445256 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7dbt\" (UniqueName: \"kubernetes.io/projected/b4fe4647-251d-45e2-9a17-283d644482b7-kube-api-access-j7dbt\") pod \"b4fe4647-251d-45e2-9a17-283d644482b7\" (UID: \"b4fe4647-251d-45e2-9a17-283d644482b7\") " Jul 21 20:23:20.447408 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.447379 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fe4647-251d-45e2-9a17-283d644482b7-kube-api-access-j7dbt" (OuterVolumeSpecName: "kube-api-access-j7dbt") pod "b4fe4647-251d-45e2-9a17-283d644482b7" (UID: "b4fe4647-251d-45e2-9a17-283d644482b7"). InnerVolumeSpecName "kube-api-access-j7dbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:23:20.506020 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.505988 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4fe4647-251d-45e2-9a17-283d644482b7" path="/var/lib/kubelet/pods/b4fe4647-251d-45e2-9a17-283d644482b7/volumes" Jul 21 20:23:20.519593 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.519566 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:20.519998 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.519985 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fe4647-251d-45e2-9a17-283d644482b7" containerName="test-curl-7441-140614109791264" Jul 21 20:23:20.520056 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.519999 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fe4647-251d-45e2-9a17-283d644482b7" containerName="test-curl-7441-140614109791264" Jul 21 20:23:20.520097 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.520063 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fe4647-251d-45e2-9a17-283d644482b7" containerName="test-curl-7441-140614109791264" Jul 21 20:23:20.523098 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.523082 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:20.529728 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.529699 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:20.545984 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.545951 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7nnq\" (UniqueName: \"kubernetes.io/projected/c160f2ae-ab8a-47c1-bfb5-4800887caa59-kube-api-access-j7nnq\") pod \"test-curl-7441-140614109792272\" (UID: \"c160f2ae-ab8a-47c1-bfb5-4800887caa59\") " pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:20.546145 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.546130 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dbt\" (UniqueName: \"kubernetes.io/projected/b4fe4647-251d-45e2-9a17-283d644482b7-kube-api-access-j7dbt\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:20.647082 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.647042 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7nnq\" (UniqueName: \"kubernetes.io/projected/c160f2ae-ab8a-47c1-bfb5-4800887caa59-kube-api-access-j7nnq\") pod \"test-curl-7441-140614109792272\" (UID: \"c160f2ae-ab8a-47c1-bfb5-4800887caa59\") " pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:20.657990 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.657965 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7nnq\" (UniqueName: \"kubernetes.io/projected/c160f2ae-ab8a-47c1-bfb5-4800887caa59-kube-api-access-j7nnq\") pod \"test-curl-7441-140614109792272\" (UID: \"c160f2ae-ab8a-47c1-bfb5-4800887caa59\") " pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:20.834267 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.834184 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:20.855950 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.855901 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" podUID="db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" containerName="main" containerID="cri-o://c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023" gracePeriod=45 Jul 21 20:23:20.971022 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:20.970993 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:20.971989 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:23:20.971961 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc160f2ae_ab8a_47c1_bfb5_4800887caa59.slice/crio-10f853f0735070ec28856577524336bfac60e470e5818d7a6ebaf5c42772fbae WatchSource:0}: Error finding container 10f853f0735070ec28856577524336bfac60e470e5818d7a6ebaf5c42772fbae: Status 404 returned error can't find the container with id 10f853f0735070ec28856577524336bfac60e470e5818d7a6ebaf5c42772fbae Jul 21 20:23:21.114672 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.114649 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:23:21.152106 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.152071 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tls-certs\") pod \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " Jul 21 20:23:21.152297 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.152118 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-dshm\") pod \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " Jul 21 20:23:21.152297 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.152179 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tmp-dir\") pod \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " Jul 21 20:23:21.152297 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.152225 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4qb2\" (UniqueName: \"kubernetes.io/projected/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-kube-api-access-g4qb2\") pod \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " Jul 21 20:23:21.152297 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.152274 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-model-cache\") pod \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " Jul 21 20:23:21.152510 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.152363 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-home\") pod \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\" (UID: \"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9\") " Jul 21 20:23:21.152766 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.152724 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" (UID: "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:23:21.152917 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.152739 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" (UID: "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:23:21.154449 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.154428 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" (UID: "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:23:21.154534 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.154425 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-home" (OuterVolumeSpecName: "home") pod "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" (UID: "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:23:21.154700 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.154678 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-dshm" (OuterVolumeSpecName: "dshm") pod "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" (UID: "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:23:21.154787 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.154696 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-kube-api-access-g4qb2" (OuterVolumeSpecName: "kube-api-access-g4qb2") pod "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" (UID: "db9f3c55-7da2-4ce7-9689-b0bfd901e2d9"). InnerVolumeSpecName "kube-api-access-g4qb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:23:21.196326 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.196294 2557 generic.go:356] "Generic (PLEG): container finished" podID="db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" containerID="c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023" exitCode=0 Jul 21 20:23:21.196495 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.196336 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" Jul 21 20:23:21.196495 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.196394 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" event={"ID":"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9","Type":"ContainerDied","Data":"c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023"} Jul 21 20:23:21.196495 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.196440 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776" event={"ID":"db9f3c55-7da2-4ce7-9689-b0bfd901e2d9","Type":"ContainerDied","Data":"e97e454f752d00e01f4c939cbb22d0601f60b2df9d05dc91ffabbed12fbb8b2d"} Jul 21 20:23:21.196495 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.196461 2557 scope.go:117] "RemoveContainer" containerID="c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023" Jul 21 20:23:21.197856 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.197745 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109791264" Jul 21 20:23:21.199125 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.199100 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109792272" event={"ID":"c160f2ae-ab8a-47c1-bfb5-4800887caa59","Type":"ContainerStarted","Data":"10f853f0735070ec28856577524336bfac60e470e5818d7a6ebaf5c42772fbae"} Jul 21 20:23:21.206213 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.206197 2557 scope.go:117] "RemoveContainer" containerID="c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023" Jul 21 20:23:21.206444 ip-10-0-134-158 kubenswrapper[2557]: E0721 20:23:21.206425 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023\": container with ID starting with c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023 not found: ID does not exist" containerID="c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023" Jul 21 20:23:21.206506 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.206452 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023"} err="failed to get container status \"c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023\": rpc error: code = NotFound desc = could not find container \"c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023\": container with ID starting with c6e5061e0d5d8363105fa932892771664f4d4725e09cb9292477aa073d9bf023 not found: ID does not exist" Jul 21 20:23:21.206506 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.206470 2557 scope.go:117] "RemoveContainer" containerID="b2e5348a1e8eb7a85d6a2d3dc2fcf79c1d3e7cb6970656df95d0bba6a540a646" Jul 21 20:23:21.219117 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.219097 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776"] Jul 21 20:23:21.224783 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.224759 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-065403/rate-test-model-065403-kserve-6c886c7899-ww776"] Jul 21 20:23:21.254013 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.253989 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tls-certs\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:21.254013 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.254010 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-dshm\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:21.254136 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.254019 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-tmp-dir\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:21.254136 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.254028 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4qb2\" (UniqueName: \"kubernetes.io/projected/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-kube-api-access-g4qb2\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:21.254136 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.254039 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-model-cache\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:21.254136 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:21.254048 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9-home\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:22.209586 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:22.209548 2557 generic.go:356] "Generic (PLEG): container finished" podID="c160f2ae-ab8a-47c1-bfb5-4800887caa59" containerID="8741a1d344235e92ad57e33a70a96eb92d363aea0c39290b84dba3caf7c1f660" exitCode=0 Jul 21 20:23:22.209974 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:22.209589 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109792272" event={"ID":"c160f2ae-ab8a-47c1-bfb5-4800887caa59","Type":"ContainerDied","Data":"8741a1d344235e92ad57e33a70a96eb92d363aea0c39290b84dba3caf7c1f660"} Jul 21 20:23:22.508146 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:22.508066 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" path="/var/lib/kubelet/pods/db9f3c55-7da2-4ce7-9689-b0bfd901e2d9/volumes" Jul 21 20:23:23.353732 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.353704 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:23.375383 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.375359 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7441-140614109792272_c160f2ae-ab8a-47c1-bfb5-4800887caa59/test-curl-7441-140614109792272/0.log" Jul 21 20:23:23.406780 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.406751 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:23.409107 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.409084 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:23.475491 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.475406 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7nnq\" (UniqueName: \"kubernetes.io/projected/c160f2ae-ab8a-47c1-bfb5-4800887caa59-kube-api-access-j7nnq\") pod \"c160f2ae-ab8a-47c1-bfb5-4800887caa59\" (UID: \"c160f2ae-ab8a-47c1-bfb5-4800887caa59\") " Jul 21 20:23:23.477498 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.477462 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c160f2ae-ab8a-47c1-bfb5-4800887caa59-kube-api-access-j7nnq" (OuterVolumeSpecName: "kube-api-access-j7nnq") pod "c160f2ae-ab8a-47c1-bfb5-4800887caa59" (UID: "c160f2ae-ab8a-47c1-bfb5-4800887caa59"). InnerVolumeSpecName "kube-api-access-j7nnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:23:23.565984 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.565953 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:23.566336 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.566323 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" containerName="main" Jul 21 20:23:23.566381 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.566338 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" containerName="main" Jul 21 20:23:23.566381 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.566347 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c160f2ae-ab8a-47c1-bfb5-4800887caa59" containerName="test-curl-7441-140614109792272" Jul 21 20:23:23.566381 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.566354 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c160f2ae-ab8a-47c1-bfb5-4800887caa59" containerName="test-curl-7441-140614109792272" Jul 21 20:23:23.566474 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.566427 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c160f2ae-ab8a-47c1-bfb5-4800887caa59" containerName="test-curl-7441-140614109792272" Jul 21 20:23:23.566474 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.566435 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="db9f3c55-7da2-4ce7-9689-b0bfd901e2d9" containerName="main" Jul 21 20:23:23.569395 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.569377 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:23.576136 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.576113 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:23.577366 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.577340 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7nnq\" (UniqueName: \"kubernetes.io/projected/c160f2ae-ab8a-47c1-bfb5-4800887caa59-kube-api-access-j7nnq\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:23.678776 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.678745 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh6xf\" (UniqueName: \"kubernetes.io/projected/86e04fa1-1b36-4df3-a8c4-4b8ad96c8776-kube-api-access-vh6xf\") pod \"test-curl-7441-140614109792272\" (UID: \"86e04fa1-1b36-4df3-a8c4-4b8ad96c8776\") " pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:23.779994 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.779898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh6xf\" (UniqueName: \"kubernetes.io/projected/86e04fa1-1b36-4df3-a8c4-4b8ad96c8776-kube-api-access-vh6xf\") pod \"test-curl-7441-140614109792272\" (UID: \"86e04fa1-1b36-4df3-a8c4-4b8ad96c8776\") " pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:23.788680 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.788655 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh6xf\" (UniqueName: \"kubernetes.io/projected/86e04fa1-1b36-4df3-a8c4-4b8ad96c8776-kube-api-access-vh6xf\") pod \"test-curl-7441-140614109792272\" (UID: \"86e04fa1-1b36-4df3-a8c4-4b8ad96c8776\") " pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:23.880089 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:23.880057 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:24.016143 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:24.016116 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:24.016771 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:23:24.016746 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e04fa1_1b36_4df3_a8c4_4b8ad96c8776.slice/crio-e21270f190f2b3caec7b1da5a2a6acb874d99ff5068f042b7d5b99296b685036 WatchSource:0}: Error finding container e21270f190f2b3caec7b1da5a2a6acb874d99ff5068f042b7d5b99296b685036: Status 404 returned error can't find the container with id e21270f190f2b3caec7b1da5a2a6acb874d99ff5068f042b7d5b99296b685036 Jul 21 20:23:24.222561 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:24.222535 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:24.222738 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:24.222581 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f853f0735070ec28856577524336bfac60e470e5818d7a6ebaf5c42772fbae" Jul 21 20:23:24.223929 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:24.223898 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109792272" event={"ID":"86e04fa1-1b36-4df3-a8c4-4b8ad96c8776","Type":"ContainerStarted","Data":"e21270f190f2b3caec7b1da5a2a6acb874d99ff5068f042b7d5b99296b685036"} Jul 21 20:23:24.236029 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:24.236002 2557 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7441-140614109792272" oldPodUID="c160f2ae-ab8a-47c1-bfb5-4800887caa59" podUID="86e04fa1-1b36-4df3-a8c4-4b8ad96c8776" Jul 21 20:23:24.505993 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:24.505965 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c160f2ae-ab8a-47c1-bfb5-4800887caa59" path="/var/lib/kubelet/pods/c160f2ae-ab8a-47c1-bfb5-4800887caa59/volumes" Jul 21 20:23:25.231421 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:25.231388 2557 generic.go:356] "Generic (PLEG): container finished" podID="86e04fa1-1b36-4df3-a8c4-4b8ad96c8776" containerID="8d75f18db64a8c51c186e6df17ecaa7bda018582202bfc1957ccb33b0a136f7b" exitCode=0 Jul 21 20:23:25.231621 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:25.231420 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7441-140614109792272" event={"ID":"86e04fa1-1b36-4df3-a8c4-4b8ad96c8776","Type":"ContainerDied","Data":"8d75f18db64a8c51c186e6df17ecaa7bda018582202bfc1957ccb33b0a136f7b"} Jul 21 20:23:26.377317 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:26.377292 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:26.398061 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:26.398030 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7441-140614109792272_86e04fa1-1b36-4df3-a8c4-4b8ad96c8776/test-curl-7441-140614109792272/0.log" Jul 21 20:23:26.423566 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:26.423534 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:26.429608 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:26.429583 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7441-140614109792272"] Jul 21 20:23:26.505997 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:26.505908 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh6xf\" (UniqueName: \"kubernetes.io/projected/86e04fa1-1b36-4df3-a8c4-4b8ad96c8776-kube-api-access-vh6xf\") pod \"86e04fa1-1b36-4df3-a8c4-4b8ad96c8776\" (UID: \"86e04fa1-1b36-4df3-a8c4-4b8ad96c8776\") " Jul 21 20:23:26.508017 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:26.507982 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e04fa1-1b36-4df3-a8c4-4b8ad96c8776-kube-api-access-vh6xf" (OuterVolumeSpecName: "kube-api-access-vh6xf") pod "86e04fa1-1b36-4df3-a8c4-4b8ad96c8776" (UID: "86e04fa1-1b36-4df3-a8c4-4b8ad96c8776"). InnerVolumeSpecName "kube-api-access-vh6xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:23:26.607219 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:26.607183 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh6xf\" (UniqueName: \"kubernetes.io/projected/86e04fa1-1b36-4df3-a8c4-4b8ad96c8776-kube-api-access-vh6xf\") on node \"ip-10-0-134-158.ec2.internal\" DevicePath \"\"" Jul 21 20:23:27.247548 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:27.247520 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7441-140614109792272" Jul 21 20:23:27.247762 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:27.247590 2557 scope.go:117] "RemoveContainer" containerID="8d75f18db64a8c51c186e6df17ecaa7bda018582202bfc1957ccb33b0a136f7b" Jul 21 20:23:28.506530 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:28.506495 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e04fa1-1b36-4df3-a8c4-4b8ad96c8776" path="/var/lib/kubelet/pods/86e04fa1-1b36-4df3-a8c4-4b8ad96c8776/volumes" Jul 21 20:23:40.095546 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:23:40.095461 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:24:02.516248 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:02.516217 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-n7rdc"] Jul 21 20:24:17.179299 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:17.179268 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-84kp2_3956c1ef-0b89-4e91-a16d-d08fa3f87476/manager/0.log" Jul 21 20:24:17.305589 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:17.305559 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tc5xz_55620973-5262-462b-bbd4-a9f22b9d6f7e/manager/0.log" Jul 21 20:24:17.593251 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:17.593226 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lzs2n_e4a6f6d5-ad74-4912-8a89-d4427ac74e23/server/0.log" Jul 21 20:24:17.727666 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:17.727640 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-k86fn_9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c/manager/0.log" Jul 21 20:24:17.852794 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:17.852708 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f74b88589-f9jgx_f2d5f7f6-4496-4a99-98da-84291973fa06/copy-manifests/0.log" Jul 21 20:24:17.859998 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:17.859976 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f74b88589-f9jgx_f2d5f7f6-4496-4a99-98da-84291973fa06/manager/0.log" Jul 21 20:24:19.721363 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:19.721330 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn_edefb881-6ed1-478a-ba17-ee65c1886005/pull/0.log" Jul 21 20:24:19.726789 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:19.726769 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn_edefb881-6ed1-478a-ba17-ee65c1886005/extract/0.log" Jul 21 20:24:19.731851 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:19.731827 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn_edefb881-6ed1-478a-ba17-ee65c1886005/util/0.log" Jul 21 20:24:19.848624 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:19.848593 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb_60086618-c52f-4e93-9514-02446c4bed24/util/0.log" Jul 21 20:24:19.854507 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:19.854488 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb_60086618-c52f-4e93-9514-02446c4bed24/pull/0.log" Jul 21 20:24:19.860756 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:19.860739 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb_60086618-c52f-4e93-9514-02446c4bed24/extract/0.log" Jul 21 20:24:19.984566 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:19.984495 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck_7a97611e-96c7-4b3d-a164-d9e8ff1cae22/util/0.log" Jul 21 20:24:19.990798 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:19.990781 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck_7a97611e-96c7-4b3d-a164-d9e8ff1cae22/pull/0.log" Jul 21 20:24:19.997228 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:19.997204 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck_7a97611e-96c7-4b3d-a164-d9e8ff1cae22/extract/0.log" Jul 21 20:24:20.115351 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:20.115321 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx_27d7fdbb-586f-4147-bcc9-68cd83a5bae9/util/0.log" Jul 21 20:24:20.122032 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:20.122011 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx_27d7fdbb-586f-4147-bcc9-68cd83a5bae9/pull/0.log" Jul 21 20:24:20.128959 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:20.128939 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx_27d7fdbb-586f-4147-bcc9-68cd83a5bae9/extract/0.log" Jul 21 20:24:20.380130 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:20.380053 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vd2sg_28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed/manager/0.log" Jul 21 20:24:20.500538 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:20.500512 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bx9bm_76cf9d8e-f047-49eb-a030-858aa3ddb7ef/manager/0.log" Jul 21 20:24:20.753958 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:20.753930 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zlwf5_bbdca730-24fc-411b-9c6a-a5b8a09a9ded/registry-server/0.log" Jul 21 20:24:20.898408 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:20.898376 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-mpl85_e0875ea3-803b-46d5-9b59-c3d713d785a2/manager/0.log" Jul 21 20:24:21.029588 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:21.029517 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-n7rdc_59728d01-a27c-4881-ad06-3c0eaaa8df75/limitador/0.log" Jul 21 20:24:21.154054 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:21.154010 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5ssp4_66329117-8064-4222-9ef2-2b96f79ebed0/manager/0.log" Jul 21 20:24:21.530356 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:21.530328 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rrvcj_2ffb7861-e586-4d4b-afe4-67165d82f0b0/istio-proxy/0.log" Jul 21 20:24:21.796825 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:21.796742 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db7f7c8d6-8b2m6_d4b5193a-3e53-426b-9ac1-7dd11d70d8e9/kube-auth-proxy/0.log" Jul 21 20:24:22.046948 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:22.046866 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-m7s7d_e417bcb3-4037-4ede-93ed-a712a90a8c70/istio-proxy/1.log" Jul 21 20:24:23.317286 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:23.317262 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-v5r5c_16fbb2f2-02fc-461d-84bd-317600ee76c5/main/0.log" Jul 21 20:24:23.431127 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:23.431099 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pwx56_f5fe1b28-bcfb-499a-977d-d01614eb3b71/main/0.log" Jul 21 20:24:29.541139 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.541108 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cvhf7/must-gather-6l54h"] Jul 21 20:24:29.541522 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.541476 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e04fa1-1b36-4df3-a8c4-4b8ad96c8776" containerName="test-curl-7441-140614109792272" Jul 21 20:24:29.541522 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.541488 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e04fa1-1b36-4df3-a8c4-4b8ad96c8776" containerName="test-curl-7441-140614109792272" Jul 21 20:24:29.541597 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.541561 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e04fa1-1b36-4df3-a8c4-4b8ad96c8776" containerName="test-curl-7441-140614109792272" Jul 21 20:24:29.544585 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.544568 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cvhf7/must-gather-6l54h" Jul 21 20:24:29.547702 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.547674 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cvhf7\"/\"openshift-service-ca.crt\"" Jul 21 20:24:29.547702 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.547674 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cvhf7\"/\"default-dockercfg-mzxhw\"" Jul 21 20:24:29.547892 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.547706 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cvhf7\"/\"kube-root-ca.crt\"" Jul 21 20:24:29.555417 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.555391 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cvhf7/must-gather-6l54h"] Jul 21 20:24:29.694730 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.694700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgqr\" (UniqueName: \"kubernetes.io/projected/412dfb63-8b99-4fe9-b7f5-32e594034afb-kube-api-access-hjgqr\") pod \"must-gather-6l54h\" (UID: \"412dfb63-8b99-4fe9-b7f5-32e594034afb\") " pod="openshift-must-gather-cvhf7/must-gather-6l54h" Jul 21 20:24:29.694929 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.694736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/412dfb63-8b99-4fe9-b7f5-32e594034afb-must-gather-output\") pod \"must-gather-6l54h\" (UID: \"412dfb63-8b99-4fe9-b7f5-32e594034afb\") " pod="openshift-must-gather-cvhf7/must-gather-6l54h" Jul 21 20:24:29.796155 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.796073 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgqr\" (UniqueName: \"kubernetes.io/projected/412dfb63-8b99-4fe9-b7f5-32e594034afb-kube-api-access-hjgqr\") pod \"must-gather-6l54h\" (UID: \"412dfb63-8b99-4fe9-b7f5-32e594034afb\") " pod="openshift-must-gather-cvhf7/must-gather-6l54h" Jul 21 20:24:29.796155 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.796109 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/412dfb63-8b99-4fe9-b7f5-32e594034afb-must-gather-output\") pod \"must-gather-6l54h\" (UID: \"412dfb63-8b99-4fe9-b7f5-32e594034afb\") " pod="openshift-must-gather-cvhf7/must-gather-6l54h" Jul 21 20:24:29.796439 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.796418 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/412dfb63-8b99-4fe9-b7f5-32e594034afb-must-gather-output\") pod \"must-gather-6l54h\" (UID: \"412dfb63-8b99-4fe9-b7f5-32e594034afb\") " pod="openshift-must-gather-cvhf7/must-gather-6l54h" Jul 21 20:24:29.806186 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.806161 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgqr\" (UniqueName: \"kubernetes.io/projected/412dfb63-8b99-4fe9-b7f5-32e594034afb-kube-api-access-hjgqr\") pod \"must-gather-6l54h\" (UID: \"412dfb63-8b99-4fe9-b7f5-32e594034afb\") " pod="openshift-must-gather-cvhf7/must-gather-6l54h" Jul 21 20:24:29.854857 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:29.854830 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cvhf7/must-gather-6l54h" Jul 21 20:24:30.045200 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:30.045168 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cvhf7/must-gather-6l54h"] Jul 21 20:24:30.055543 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:24:30.055518 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod412dfb63_8b99_4fe9_b7f5_32e594034afb.slice/crio-fd961ff245bffc3e0dbb0263d39e8fe4e6fecec6fd315e7b40b1924cfe3362a0 WatchSource:0}: Error finding container fd961ff245bffc3e0dbb0263d39e8fe4e6fecec6fd315e7b40b1924cfe3362a0: Status 404 returned error can't find the container with id fd961ff245bffc3e0dbb0263d39e8fe4e6fecec6fd315e7b40b1924cfe3362a0 Jul 21 20:24:30.601301 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:30.601264 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cvhf7/must-gather-6l54h" event={"ID":"412dfb63-8b99-4fe9-b7f5-32e594034afb","Type":"ContainerStarted","Data":"fd961ff245bffc3e0dbb0263d39e8fe4e6fecec6fd315e7b40b1924cfe3362a0"} Jul 21 20:24:31.611371 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:31.611328 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cvhf7/must-gather-6l54h" event={"ID":"412dfb63-8b99-4fe9-b7f5-32e594034afb","Type":"ContainerStarted","Data":"c49584e716e227171434698e775f2b501b05c9565b038ed4754c6bdd50847408"} Jul 21 20:24:31.611757 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:31.611385 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cvhf7/must-gather-6l54h" event={"ID":"412dfb63-8b99-4fe9-b7f5-32e594034afb","Type":"ContainerStarted","Data":"aeac50e4c2416bb0a43b7bc72c2f0c492da262863c09af9ae6963f4771c68380"} Jul 21 20:24:31.628480 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:31.628429 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cvhf7/must-gather-6l54h" podStartSLOduration=1.9842207360000002 podStartE2EDuration="2.628412409s" totalImagesPullingTime="644.191673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:24:31.627605971 +0000 UTC m=+2834.812047653" watchObservedRunningTime="2026-07-21 20:24:31.628412409 +0000 UTC m=+2834.812854090" Jul 21 20:24:32.235967 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:32.235936 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kd25k_aba01382-69a9-4443-b1a2-fe15605e698d/global-pull-secret-syncer/0.log" Jul 21 20:24:32.402727 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:32.402700 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jqq6h_b96f04b2-899b-4860-8654-6e0061a4d0dc/konnectivity-agent/0.log" Jul 21 20:24:32.491312 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:32.491241 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-158.ec2.internal_bfae3c7bef61f9e9cdfc141005768ce6/haproxy/0.log" Jul 21 20:24:36.660462 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.660424 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn_edefb881-6ed1-478a-ba17-ee65c1886005/extract/0.log" Jul 21 20:24:36.681023 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.680995 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn_edefb881-6ed1-478a-ba17-ee65c1886005/util/0.log" Jul 21 20:24:36.706216 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.706182 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598cgxn_edefb881-6ed1-478a-ba17-ee65c1886005/pull/0.log" Jul 21 20:24:36.733726 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.733690 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb_60086618-c52f-4e93-9514-02446c4bed24/extract/0.log" Jul 21 20:24:36.763340 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.763306 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb_60086618-c52f-4e93-9514-02446c4bed24/util/0.log" Jul 21 20:24:36.798893 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.798686 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrddb_60086618-c52f-4e93-9514-02446c4bed24/pull/0.log" Jul 21 20:24:36.835132 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.835099 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck_7a97611e-96c7-4b3d-a164-d9e8ff1cae22/extract/0.log" Jul 21 20:24:36.853854 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.853691 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck_7a97611e-96c7-4b3d-a164-d9e8ff1cae22/util/0.log" Jul 21 20:24:36.872559 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.872383 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lzgck_7a97611e-96c7-4b3d-a164-d9e8ff1cae22/pull/0.log" Jul 21 20:24:36.901285 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.901253 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx_27d7fdbb-586f-4147-bcc9-68cd83a5bae9/extract/0.log" Jul 21 20:24:36.923331 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.923251 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx_27d7fdbb-586f-4147-bcc9-68cd83a5bae9/util/0.log" Jul 21 20:24:36.945818 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:36.945787 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpbfx_27d7fdbb-586f-4147-bcc9-68cd83a5bae9/pull/0.log" Jul 21 20:24:37.296746 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:37.296641 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vd2sg_28d3d2cd-aa2a-4e59-bcb0-e1f38ecb3aed/manager/0.log" Jul 21 20:24:37.326348 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:37.326319 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bx9bm_76cf9d8e-f047-49eb-a030-858aa3ddb7ef/manager/0.log" Jul 21 20:24:37.417051 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:37.417011 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zlwf5_bbdca730-24fc-411b-9c6a-a5b8a09a9ded/registry-server/0.log" Jul 21 20:24:37.627537 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:37.627412 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-mpl85_e0875ea3-803b-46d5-9b59-c3d713d785a2/manager/0.log" Jul 21 20:24:37.647285 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:37.647259 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-n7rdc_59728d01-a27c-4881-ad06-3c0eaaa8df75/limitador/0.log" Jul 21 20:24:37.726490 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:37.726058 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5ssp4_66329117-8064-4222-9ef2-2b96f79ebed0/manager/0.log" Jul 21 20:24:39.647577 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:39.647537 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-58ffcbfbc7-6hw54_a77c31f0-8c2f-4a2e-b716-bfa10cd731e1/metrics-server/0.log" Jul 21 20:24:39.708998 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:39.708967 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f4f4w_e2ee63bc-ee4d-4646-b833-b57fe5467935/node-exporter/0.log" Jul 21 20:24:39.725220 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:39.725181 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f4f4w_e2ee63bc-ee4d-4646-b833-b57fe5467935/kube-rbac-proxy/0.log" Jul 21 20:24:39.742029 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:39.741958 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f4f4w_e2ee63bc-ee4d-4646-b833-b57fe5467935/init-textfile/0.log" Jul 21 20:24:40.152189 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:40.152150 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-98k99_0eef2534-1587-4685-9062-a783e3b7ed3e/prometheus-operator/0.log" Jul 21 20:24:40.172185 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:40.172113 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-98k99_0eef2534-1587-4685-9062-a783e3b7ed3e/kube-rbac-proxy/0.log" Jul 21 20:24:40.192932 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:40.192887 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-kqpcx_d14e31a3-1f62-489d-9d7a-32bb322e155c/prometheus-operator-admission-webhook/0.log" Jul 21 20:24:41.581646 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:41.581616 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-q4sp7_3eb90b32-fc9a-4523-8348-cc6f777f6ac9/networking-console-plugin/0.log" Jul 21 20:24:42.192708 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.192677 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/1.log" Jul 21 20:24:42.198779 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.198753 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fvv9d_639481df-2652-4b1d-b003-c6aef4eb44af/console-operator/2.log" Jul 21 20:24:42.550457 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.550374 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv"] Jul 21 20:24:42.555464 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.555436 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.569195 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.569167 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv"] Jul 21 20:24:42.639013 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.637667 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-sys\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.639013 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.637749 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-proc\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.639013 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.637796 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7cc\" (UniqueName: \"kubernetes.io/projected/dbf7eda0-9b04-478c-88f0-99b95d7854a5-kube-api-access-lg7cc\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.639013 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.637847 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-lib-modules\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.639013 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.637901 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-podres\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.738980 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.738942 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-proc\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.739171 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.739019 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7cc\" (UniqueName: \"kubernetes.io/projected/dbf7eda0-9b04-478c-88f0-99b95d7854a5-kube-api-access-lg7cc\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.739171 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.739072 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-lib-modules\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.739171 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.739065 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-proc\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.739171 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.739153 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-podres\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.739399 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.739278 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-podres\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.739399 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.739287 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-sys\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.739399 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.739336 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-sys\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.739602 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.739578 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbf7eda0-9b04-478c-88f0-99b95d7854a5-lib-modules\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.747420 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.747386 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7cc\" (UniqueName: \"kubernetes.io/projected/dbf7eda0-9b04-478c-88f0-99b95d7854a5-kube-api-access-lg7cc\") pod \"perf-node-gather-daemonset-2gwqv\" (UID: \"dbf7eda0-9b04-478c-88f0-99b95d7854a5\") " pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:42.750294 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.750272 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-99458d98-7mkt2_0ec39da6-4a11-4bae-8264-bbbf43286441/console/0.log" Jul 21 20:24:42.869518 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:42.869435 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:43.059868 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:43.059659 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv"] Jul 21 20:24:43.062301 ip-10-0-134-158 kubenswrapper[2557]: W0721 20:24:43.062270 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddbf7eda0_9b04_478c_88f0_99b95d7854a5.slice/crio-dda69b8949bedec734e9405d2a6ad41b5945783c9ada97d77b1928f5c0850fc7 WatchSource:0}: Error finding container dda69b8949bedec734e9405d2a6ad41b5945783c9ada97d77b1928f5c0850fc7: Status 404 returned error can't find the container with id dda69b8949bedec734e9405d2a6ad41b5945783c9ada97d77b1928f5c0850fc7 Jul 21 20:24:43.713918 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:43.713856 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" event={"ID":"dbf7eda0-9b04-478c-88f0-99b95d7854a5","Type":"ContainerStarted","Data":"5c9fae44339bae4492ecbadbfcb734c6e570e1b30cea2dce5f32c2bb25e88daa"} Jul 21 20:24:43.714371 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:43.713963 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:43.714371 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:43.713991 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" event={"ID":"dbf7eda0-9b04-478c-88f0-99b95d7854a5","Type":"ContainerStarted","Data":"dda69b8949bedec734e9405d2a6ad41b5945783c9ada97d77b1928f5c0850fc7"} Jul 21 20:24:44.492794 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:44.492745 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dhqw9_2c1b84bc-9f08-49b0-b538-c09d743953bf/dns/0.log" Jul 21 20:24:44.510484 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:44.510459 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dhqw9_2c1b84bc-9f08-49b0-b538-c09d743953bf/kube-rbac-proxy/0.log" Jul 21 20:24:44.641338 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:44.641309 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-djwk4_02c2a5d8-4576-45b1-9c2a-ddbd5012322b/dns-node-resolver/0.log" Jul 21 20:24:45.195325 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:45.195294 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7744fff668-gkfr6_87f3f5fc-bd83-40b6-b4cf-7a106e334aee/registry/0.log" Jul 21 20:24:45.215211 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:45.215182 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-64pkq_0e3f4244-32a9-467b-b77c-26efe45e6aac/node-ca/0.log" Jul 21 20:24:46.171129 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:46.171091 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rrvcj_2ffb7861-e586-4d4b-afe4-67165d82f0b0/istio-proxy/0.log" Jul 21 20:24:46.417079 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:46.417049 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db7f7c8d6-8b2m6_d4b5193a-3e53-426b-9ac1-7dd11d70d8e9/kube-auth-proxy/0.log" Jul 21 20:24:46.468318 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:46.468244 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-m7s7d_e417bcb3-4037-4ede-93ed-a712a90a8c70/istio-proxy/0.log" Jul 21 20:24:46.502698 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:46.502644 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-m7s7d_e417bcb3-4037-4ede-93ed-a712a90a8c70/istio-proxy/1.log" Jul 21 20:24:47.290129 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:47.290100 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-czf26_51ed889a-0a84-4523-8c9b-989d2ac767fe/serve-healthcheck-canary/0.log" Jul 21 20:24:47.841714 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:47.841678 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-kcl4p_58826593-b4e6-4152-a0a9-a5b9746af2a9/insights-operator/0.log" Jul 21 20:24:47.842327 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:47.842310 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-kcl4p_58826593-b4e6-4152-a0a9-a5b9746af2a9/insights-operator/1.log" Jul 21 20:24:47.865050 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:47.865021 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mvk77_16431569-e917-4a3e-a020-0459e9a3c984/kube-rbac-proxy/0.log" Jul 21 20:24:47.888061 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:47.888039 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mvk77_16431569-e917-4a3e-a020-0459e9a3c984/exporter/0.log" Jul 21 20:24:47.918387 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:47.918358 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mvk77_16431569-e917-4a3e-a020-0459e9a3c984/extractor/0.log" Jul 21 20:24:50.218720 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:50.218674 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-84kp2_3956c1ef-0b89-4e91-a16d-d08fa3f87476/manager/0.log" Jul 21 20:24:50.466214 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:50.466170 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tc5xz_55620973-5262-462b-bbd4-a9f22b9d6f7e/manager/0.log" Jul 21 20:24:50.676277 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:50.676253 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lzs2n_e4a6f6d5-ad74-4912-8a89-d4427ac74e23/server/0.log" Jul 21 20:24:50.741412 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:50.741384 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" Jul 21 20:24:50.764260 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:50.764186 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cvhf7/perf-node-gather-daemonset-2gwqv" podStartSLOduration=8.764165737 podStartE2EDuration="8.764165737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:24:43.734045601 +0000 UTC m=+2846.918487310" watchObservedRunningTime="2026-07-21 20:24:50.764165737 +0000 UTC m=+2853.948607418" Jul 21 20:24:50.795516 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:50.795478 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-k86fn_9dc7e64e-9e32-4a8c-a3a4-a30c6bfb823c/manager/0.log" Jul 21 20:24:50.821704 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:50.821661 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f74b88589-f9jgx_f2d5f7f6-4496-4a99-98da-84291973fa06/manager/0.log" Jul 21 20:24:50.847968 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:50.847940 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f74b88589-f9jgx_f2d5f7f6-4496-4a99-98da-84291973fa06/copy-manifests/0.log" Jul 21 20:24:52.611528 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:52.611495 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rrvcj_2ffb7861-e586-4d4b-afe4-67165d82f0b0/istio-proxy/0.log" Jul 21 20:24:52.945504 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:52.945476 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db7f7c8d6-8b2m6_d4b5193a-3e53-426b-9ac1-7dd11d70d8e9/kube-auth-proxy/0.log" Jul 21 20:24:52.991383 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:52.991330 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-m7s7d_e417bcb3-4037-4ede-93ed-a712a90a8c70/istio-proxy/0.log" Jul 21 20:24:53.029895 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:53.029854 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-m7s7d_e417bcb3-4037-4ede-93ed-a712a90a8c70/istio-proxy/1.log" Jul 21 20:24:54.399958 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:54.399926 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rrvcj_2ffb7861-e586-4d4b-afe4-67165d82f0b0/istio-proxy/0.log" Jul 21 20:24:54.634137 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:54.634103 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db7f7c8d6-8b2m6_d4b5193a-3e53-426b-9ac1-7dd11d70d8e9/kube-auth-proxy/0.log" Jul 21 20:24:54.673454 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:54.673373 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-m7s7d_e417bcb3-4037-4ede-93ed-a712a90a8c70/istio-proxy/0.log" Jul 21 20:24:54.707649 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:54.707620 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-m7s7d_e417bcb3-4037-4ede-93ed-a712a90a8c70/istio-proxy/1.log" Jul 21 20:24:58.128968 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:58.128931 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-7k4hz_ac97b694-bcc5-47c3-a779-9bf9b79bbdf0/kube-storage-version-migrator-operator/1.log" Jul 21 20:24:58.129849 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:58.129822 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-7k4hz_ac97b694-bcc5-47c3-a779-9bf9b79bbdf0/kube-storage-version-migrator-operator/0.log" Jul 21 20:24:59.498653 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.498625 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lbfw_66af5356-d22c-46a3-972b-76f5050acfee/kube-multus-additional-cni-plugins/0.log" Jul 21 20:24:59.522370 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.522344 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lbfw_66af5356-d22c-46a3-972b-76f5050acfee/egress-router-binary-copy/0.log" Jul 21 20:24:59.546595 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.546567 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lbfw_66af5356-d22c-46a3-972b-76f5050acfee/cni-plugins/0.log" Jul 21 20:24:59.565407 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.565376 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lbfw_66af5356-d22c-46a3-972b-76f5050acfee/bond-cni-plugin/0.log" Jul 21 20:24:59.590175 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.590153 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lbfw_66af5356-d22c-46a3-972b-76f5050acfee/routeoverride-cni/0.log" Jul 21 20:24:59.610166 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.610144 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lbfw_66af5356-d22c-46a3-972b-76f5050acfee/whereabouts-cni-bincopy/0.log" Jul 21 20:24:59.628673 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.628645 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8lbfw_66af5356-d22c-46a3-972b-76f5050acfee/whereabouts-cni/0.log" Jul 21 20:24:59.701927 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.701896 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x26r8_265c7b96-c23f-4084-8ff8-fa2789f82cee/kube-multus/0.log" Jul 21 20:24:59.724198 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.724173 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fjfkl_30e72480-263d-4449-87d6-4a30fdbcfcee/network-metrics-daemon/0.log" Jul 21 20:24:59.750646 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:24:59.750578 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fjfkl_30e72480-263d-4449-87d6-4a30fdbcfcee/kube-rbac-proxy/0.log" Jul 21 20:25:00.963031 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:25:00.963006 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn8h5_462d9c54-6e71-424a-9489-166c81de167a/ovn-controller/0.log" Jul 21 20:25:00.977445 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:25:00.977419 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn8h5_462d9c54-6e71-424a-9489-166c81de167a/ovn-acl-logging/0.log" Jul 21 20:25:00.998547 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:25:00.998521 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn8h5_462d9c54-6e71-424a-9489-166c81de167a/kube-rbac-proxy-node/0.log" Jul 21 20:25:01.021586 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:25:01.021558 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn8h5_462d9c54-6e71-424a-9489-166c81de167a/kube-rbac-proxy-ovn-metrics/0.log" Jul 21 20:25:01.040286 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:25:01.040263 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn8h5_462d9c54-6e71-424a-9489-166c81de167a/northd/0.log" Jul 21 20:25:01.059219 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:25:01.059190 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn8h5_462d9c54-6e71-424a-9489-166c81de167a/nbdb/0.log" Jul 21 20:25:01.078851 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:25:01.078824 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn8h5_462d9c54-6e71-424a-9489-166c81de167a/sbdb/0.log" Jul 21 20:25:01.146213 ip-10-0-134-158 kubenswrapper[2557]: I0721 20:25:01.146185 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bn8h5_462d9c54-6e71-424a-9489-166c81de167a/ovnkube-controller/0.log"