Jul 06 19:22:55.787976 ip-10-0-140-103 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:22:56.164262 ip-10-0-140-103 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:22:56.164262 ip-10-0-140-103 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:22:56.164262 ip-10-0-140-103 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:22:56.164262 ip-10-0-140-103 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:22:56.164262 ip-10-0-140-103 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:22:56.164944 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.164333 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:22:56.167894 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167876 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:56.167894 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167894 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167898 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167902 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167905 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167908 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167911 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167914 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167917 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167921 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167923 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167926 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167929 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167933 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167935 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167938 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167941 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167943 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167947 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167951 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:56.167959 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167954 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167957 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167960 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167963 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167966 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167969 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167972 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167975 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167977 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167980 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167983 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167986 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167989 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.167993 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168002 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168007 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168010 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168013 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168016 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168019 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:56.168478 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168022 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168024 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168027 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168030 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168033 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168035 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168038 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168041 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168045 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168049 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168051 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168054 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168057 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168059 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168063 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168065 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168068 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168070 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168073 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168076 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:56.169199 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168078 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168080 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168083 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168086 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168088 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168091 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168093 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168096 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168098 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168100 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168103 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168105 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168107 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168110 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168112 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168116 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168118 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168121 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168123 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168126 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:56.169725 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168128 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168131 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168134 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168136 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168139 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168141 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168145 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168147 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168150 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168152 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168155 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168158 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168160 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168163 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168165 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168168 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168170 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168173 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:56.170244 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.168175 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:56.171214 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171185 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171217 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171222 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171225 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171228 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171231 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171234 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171236 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171239 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171241 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171244 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171247 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171250 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171252 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171255 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:56.171250 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171258 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171261 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171264 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171267 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171269 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171272 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171274 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171276 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171279 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171281 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171284 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171286 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171290 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171293 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171295 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171297 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171300 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171303 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171305 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171308 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:56.171617 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171310 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171312 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171315 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171320 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171324 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171327 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171329 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171332 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171348 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171351 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171354 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171357 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171359 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171363 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171366 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171369 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171371 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171376 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171380 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:56.172112 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171382 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171385 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171388 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171390 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171393 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171396 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171399 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171402 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171404 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171406 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171414 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171416 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171419 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171421 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171424 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171426 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171429 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171431 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171434 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171437 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:56.172587 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171440 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171442 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171446 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171449 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171451 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171453 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171456 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171458 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171463 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171467 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171470 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171473 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171476 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171478 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171481 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171484 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171486 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171488 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171491 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:56.173070 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171494 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171496 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171499 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171502 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171505 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171507 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171944 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171950 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171953 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171956 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171959 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171961 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171965 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171968 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171970 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171973 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171975 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171978 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171981 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171983 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:56.173543 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171986 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171988 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171992 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.171995 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172008 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172012 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172014 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172017 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172019 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172022 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172025 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172027 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172032 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172036 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172039 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172043 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172045 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172048 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172050 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:56.174047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172053 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172056 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172058 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172060 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172063 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172066 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172069 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172071 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172074 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172076 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172079 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172082 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172084 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172086 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172089 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172092 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172096 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172098 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172101 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172103 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:56.174530 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172106 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172109 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172112 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172115 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172118 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172120 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172123 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172126 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172129 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172131 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172134 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172137 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172139 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172142 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172145 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172149 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172152 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172156 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172159 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:56.175005 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172162 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172164 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172167 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172169 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172172 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172174 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172177 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172179 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172182 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172185 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172188 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172190 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172193 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172195 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172198 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172200 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172217 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172220 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172223 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172227 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:56.175500 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172230 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172233 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172236 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172239 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172241 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172244 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.172247 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172328 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172339 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172375 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172380 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172385 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172388 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172393 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172397 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172400 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172403 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172407 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172410 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172414 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172416 2573 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172419 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172422 2573 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:22:56.175986 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172427 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172430 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172434 2573 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172437 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172440 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172443 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172447 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172451 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172454 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172457 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172460 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172463 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172466 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172469 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172473 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172476 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172480 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172483 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172486 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172489 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172492 2573 flags.go:64] FLAG: --enable-server="true" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172495 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172500 2573 flags.go:64] FLAG: --event-burst="100" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172503 2573 flags.go:64] FLAG: --event-qps="50" Jul 06 19:22:56.176564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172506 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172509 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172512 2573 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172516 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172519 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172522 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172525 2573 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172528 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172531 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172538 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172541 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172544 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172547 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172550 2573 flags.go:64] FLAG: --feature-gates="" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172553 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172557 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172560 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172563 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172568 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172571 2573 flags.go:64] FLAG: --help="false" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172574 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172577 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172580 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:22:56.177164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172583 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172586 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172590 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172593 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172596 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172600 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172605 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172609 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172614 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172617 2573 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172620 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172623 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172626 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172629 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172632 2573 flags.go:64] FLAG: --lock-file="" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172634 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172637 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172640 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172646 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172650 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172652 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172655 2573 flags.go:64] FLAG: --logging-format="text" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172658 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172661 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:22:56.177741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172664 2573 flags.go:64] FLAG: --manifest-url="" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172667 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172671 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172676 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172680 2573 flags.go:64] FLAG: --max-pods="110" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172683 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172686 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172689 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172692 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172695 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172698 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172701 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172709 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172713 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172716 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172719 2573 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172721 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172727 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172729 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172732 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172735 2573 flags.go:64] FLAG: --port="10250" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172738 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172743 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0786e7d92de833e5d" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172748 2573 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:22:56.178348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172753 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172756 2573 flags.go:64] FLAG: --register-node="true" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172759 2573 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172762 2573 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172765 2573 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172768 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172771 2573 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172775 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172778 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172781 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172783 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172787 2573 flags.go:64] FLAG: --runonce="false" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172790 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172793 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172796 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172810 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172814 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172818 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172821 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172824 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172827 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172830 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172833 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172836 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172839 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:22:56.178965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172842 2573 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172844 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172850 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172853 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172856 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172860 2573 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172863 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172865 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172868 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172871 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172875 2573 flags.go:64] FLAG: --v="2" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172879 2573 flags.go:64] FLAG: --version="false" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172884 2573 flags.go:64] FLAG: --vmodule="" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172888 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.172891 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173016 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173022 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173025 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173030 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173035 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173038 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173041 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173044 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:56.180047 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173046 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173049 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173052 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173054 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173057 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173060 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173063 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173066 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173069 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173072 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173074 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173077 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173080 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173083 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173086 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173088 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173091 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173094 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173096 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173099 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:56.181075 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173101 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173104 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173107 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173110 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173112 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173115 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173117 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173121 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173125 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173127 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173130 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173133 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173135 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173138 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173140 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173143 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173145 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173148 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173150 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173153 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:56.181946 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173156 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173158 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173161 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173163 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173166 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173169 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173172 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173176 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173179 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173182 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173185 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173188 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173191 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173193 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173196 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173199 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173217 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173222 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173226 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:56.183007 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173232 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173236 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173239 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173242 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173246 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173249 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173252 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173254 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173257 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173259 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173262 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173264 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173267 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173269 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173272 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173274 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173277 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173280 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173282 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173285 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:56.183683 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173287 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173290 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173292 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173294 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173297 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173301 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173304 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173308 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173311 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173314 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173317 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.173319 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.173956 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.181218 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.181242 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:22:56.184537 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.181277 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.181285 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181373 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181383 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181390 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181396 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181400 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181405 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181410 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181414 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181419 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181423 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181427 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181432 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181439 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181445 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181450 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181454 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181458 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:56.185003 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181463 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181467 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181471 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181476 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181480 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181484 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181489 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181493 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181500 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181506 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181511 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181516 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181522 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181528 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181532 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181537 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181541 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181545 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181550 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181554 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:56.185602 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181558 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181562 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181566 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181570 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181574 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181578 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181583 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181587 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181591 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181596 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181600 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181604 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181608 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181612 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181616 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181620 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181624 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181629 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181633 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181637 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:56.186131 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181641 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181645 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181649 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181654 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181659 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181664 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181668 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181673 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181677 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181681 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181685 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181689 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181693 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181697 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181702 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181706 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181710 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181714 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181719 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:56.186645 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181723 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181727 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181731 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181736 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181740 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181744 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181748 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181753 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181757 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181761 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181765 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181769 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181773 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181777 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181781 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181786 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181790 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181795 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181801 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:56.187170 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181808 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181813 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181818 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.181822 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.181831 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182013 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182021 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182026 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182031 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182036 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182041 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182045 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182049 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182054 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182058 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:56.187716 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182063 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182068 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182073 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182078 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182083 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182089 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182095 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182099 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182104 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182108 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182113 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182118 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182122 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182126 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182131 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182135 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182140 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182144 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182151 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:56.188135 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182155 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182159 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182163 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182167 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182171 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182175 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182179 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182183 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182187 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182192 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182196 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182200 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182222 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182227 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182230 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182236 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182240 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182244 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182248 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182253 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:56.188695 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182256 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182260 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182264 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182268 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182271 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182275 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182279 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182283 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182286 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182291 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182295 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182301 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182306 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182311 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182315 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182319 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182323 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182326 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182330 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182334 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:56.189487 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182338 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182342 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182346 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182350 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182353 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182357 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182361 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182365 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182369 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182373 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182377 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182381 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182388 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182392 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182397 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182402 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182407 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182411 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182416 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:56.190312 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182421 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182425 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182429 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182434 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182438 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182442 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182448 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182452 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182457 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182461 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:56.182465 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.182473 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.182651 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.185310 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:22:56.191066 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.186701 2573 server.go:1019] "Starting client certificate rotation" Jul 06 19:22:56.191548 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.186810 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:22:56.191548 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.186851 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:22:56.191548 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.189443 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:22:56.191548 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.191351 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:22:56.202292 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.202270 2573 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:22:56.204216 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.204185 2573 log.go:25] "Validated CRI v1 image API" Jul 06 19:22:56.205096 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.205074 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:22:56.208065 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.208042 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f2ad5da1-0628-482d-969b-b24d0d465467:/dev/nvme0n1p4 fdbbd467-f535-4157-86e7-51bfd065dda5:/dev/nvme0n1p3] Jul 06 19:22:56.208145 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.208062 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:22:56.214313 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.214177 2573 manager.go:216] Machine: {Timestamp:2026-07-06 19:22:56.211760556 +0000 UTC m=+0.308988302 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3081080 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec242ddf03285590e8937bafc1a74ed7 SystemUUID:ec242ddf-0328-5590-e893-7bafc1a74ed7 BootID:0457ba30-d5f0-4270-98ea-fec270b97b2c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:96:86:51:bf:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:96:86:51:bf:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:34:2b:dd:64:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:22:56.214313 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.214307 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:22:56.214427 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.214408 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:22:56.217596 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.217564 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:22:56.217760 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.217601 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:22:56.218802 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.218790 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:22:56.218844 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.218806 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:22:56.218844 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.218814 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:22:56.219439 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.219427 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:22:56.220681 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.220670 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:22:56.221698 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.221681 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:22:56.223656 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.223644 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:22:56.223774 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.223765 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:22:56.223819 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.223802 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:22:56.223853 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.223820 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:22:56.223853 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.223834 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:22:56.223853 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.223845 2573 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:22:56.223853 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.223854 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:22:56.224271 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.224250 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:22:56.224369 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.224276 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:22:56.226373 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.226358 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:22:56.228115 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.228103 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:22:56.228176 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.228133 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:22:56.229188 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229177 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:22:56.229235 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229194 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:22:56.229235 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229200 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:22:56.229235 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229221 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:22:56.229320 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229239 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:22:56.229320 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229246 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:22:56.229320 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229254 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:22:56.229320 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229259 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:22:56.229320 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229265 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:22:56.229320 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229270 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:22:56.229320 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229282 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:22:56.229320 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.229287 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:22:56.230638 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.230628 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:22:56.230638 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.230638 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:22:56.233868 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.233857 2573 server.go:1268] "Started kubelet" Jul 06 19:22:56.234038 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.234003 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:22:56.234142 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.233991 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:22:56.234142 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.234091 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:22:56.234902 ip-10-0-140-103 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:22:56.239877 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.239818 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:22:56.240034 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.240018 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-103.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:22:56.240089 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.240014 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:22:56.240199 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.240172 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:22:56.241065 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.241040 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:22:56.242584 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.242564 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:22:56.244010 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.243060 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-103.ec2.internal.18bfc9b98d019885 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-103.ec2.internal,UID:ip-10-0-140-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-103.ec2.internal,},FirstTimestamp:2026-07-06 19:22:56.233830533 +0000 UTC m=+0.331058279,LastTimestamp:2026-07-06 19:22:56.233830533 +0000 UTC m=+0.331058279,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-103.ec2.internal,}" Jul 06 19:22:56.251324 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.251305 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:22:56.251494 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.251308 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:22:56.251965 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.251949 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:56.251965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.251959 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:22:56.252082 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.251976 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:22:56.252170 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252148 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:22:56.252170 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252165 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:22:56.252170 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252174 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:22:56.252443 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252426 2573 factory.go:55] Registering systemd factory Jul 06 19:22:56.252443 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252445 2573 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:22:56.252690 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252676 2573 factory.go:153] Registering CRI-O factory Jul 06 19:22:56.252690 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252690 2573 factory.go:223] Registration of the crio container factory successfully Jul 06 19:22:56.252816 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252742 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:22:56.252816 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252766 2573 factory.go:103] Registering Raw factory Jul 06 19:22:56.252816 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.252783 2573 manager.go:1196] Started watching for new ooms in manager Jul 06 19:22:56.253253 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.253238 2573 manager.go:349] Starting recovery of all containers Jul 06 19:22:56.254104 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.253865 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:22:56.262978 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.262952 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwd9x" Jul 06 19:22:56.264125 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.264076 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:22:56.264638 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.264621 2573 manager.go:354] Recovery completed Jul 06 19:22:56.268908 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.268896 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:56.271072 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.271053 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:56.271163 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.271088 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:56.271163 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.271104 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:56.271783 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.271769 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:22:56.271783 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.271781 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:22:56.271917 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.271799 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:22:56.271917 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.271829 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:56.272709 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.272679 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:56.272786 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.272724 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:56.272786 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.272739 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:56.273422 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.273408 2573 policy_none.go:49] "None policy: Start" Jul 06 19:22:56.273485 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.273429 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:22:56.273485 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.273444 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:22:56.274600 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.274587 2573 policy_none.go:47] "Start" Jul 06 19:22:56.281054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.280001 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwd9x" Jul 06 19:22:56.311068 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.311043 2573 manager.go:337] "Starting Device Plugin manager" Jul 06 19:22:56.311177 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.311095 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:22:56.311177 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.311111 2573 server.go:87] "Starting device plugin registration server" Jul 06 19:22:56.311468 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.311452 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:22:56.311537 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.311469 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:22:56.311612 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.311579 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:22:56.311700 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.311681 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:22:56.311700 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.311693 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:22:56.312339 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.312312 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:22:56.312413 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.312382 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:56.386630 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.386593 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:22:56.388048 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.387829 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:22:56.388048 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.387864 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:22:56.388048 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.387889 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:22:56.388048 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.387925 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:22:56.391145 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.391122 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:56.412624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.412595 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:56.414030 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.414014 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:56.414134 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.414047 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:56.414134 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.414063 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:56.414134 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.414090 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.423144 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.423092 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.423144 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.423117 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-103.ec2.internal\": node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:56.435405 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.435380 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:56.488301 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.488241 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal"] Jul 06 19:22:56.488482 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.488354 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:56.489502 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.489480 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:56.489585 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.489531 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:56.489585 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.489549 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:56.492836 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.492821 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:56.492976 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.492959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.493029 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.492999 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:56.493786 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.493752 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:56.493928 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.493807 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:56.493928 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.493753 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:56.493928 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.493849 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:56.493928 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.493867 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:56.493928 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.493881 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:56.495166 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.495149 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.495268 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.495175 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:56.495991 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.495972 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:56.496090 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.496008 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:56.496090 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.496023 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:56.520405 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.520381 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-103.ec2.internal\" not found" node="ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.525205 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.525185 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-103.ec2.internal\" not found" node="ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.535858 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.535837 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:56.554416 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.554387 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76adf7ddaf5e685a8613fa423bc7c3be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"76adf7ddaf5e685a8613fa423bc7c3be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.554569 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.554423 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76adf7ddaf5e685a8613fa423bc7c3be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"76adf7ddaf5e685a8613fa423bc7c3be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.554569 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.554442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce19eea5e92a6ee78e17d806a018297b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-103.ec2.internal\" (UID: \"ce19eea5e92a6ee78e17d806a018297b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.635952 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.635910 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:56.655178 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.655138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76adf7ddaf5e685a8613fa423bc7c3be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"76adf7ddaf5e685a8613fa423bc7c3be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.655319 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.655185 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76adf7ddaf5e685a8613fa423bc7c3be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"76adf7ddaf5e685a8613fa423bc7c3be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.655319 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.655214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce19eea5e92a6ee78e17d806a018297b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-103.ec2.internal\" (UID: \"ce19eea5e92a6ee78e17d806a018297b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.655319 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.655298 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76adf7ddaf5e685a8613fa423bc7c3be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"76adf7ddaf5e685a8613fa423bc7c3be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.655449 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.655342 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76adf7ddaf5e685a8613fa423bc7c3be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal\" (UID: \"76adf7ddaf5e685a8613fa423bc7c3be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.655449 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.655367 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce19eea5e92a6ee78e17d806a018297b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-103.ec2.internal\" (UID: \"ce19eea5e92a6ee78e17d806a018297b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.737201 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.737114 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:56.829704 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.829650 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.829832 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:56.829740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" Jul 06 19:22:56.838185 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.838164 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:56.941256 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:56.941206 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:57.047916 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:57.047826 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:57.079233 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.079203 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:57.148341 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:57.148304 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:57.187523 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.187481 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:22:57.188312 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.187691 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:22:57.188312 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.187691 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:22:57.248657 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:57.248620 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:57.283657 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.283606 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:17:57 +0000 UTC" deadline="2028-03-21 13:24:45.595214558 +0000 UTC" Jul 06 19:22:57.283657 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.283645 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14970h1m48.311573011s" Jul 06 19:22:57.349379 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:57.349344 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:57.435760 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:57.435644 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76adf7ddaf5e685a8613fa423bc7c3be.slice/crio-56479e0793cc327995dff15c684ac4edc4b21a56ccfec266f9aa949697a1d6c3 WatchSource:0}: Error finding container 56479e0793cc327995dff15c684ac4edc4b21a56ccfec266f9aa949697a1d6c3: Status 404 returned error can't find the container with id 56479e0793cc327995dff15c684ac4edc4b21a56ccfec266f9aa949697a1d6c3 Jul 06 19:22:57.437008 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:57.436983 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce19eea5e92a6ee78e17d806a018297b.slice/crio-d3837bc37806610434e546b5dea5e4a80c407804a60940b8304335429ede3015 WatchSource:0}: Error finding container d3837bc37806610434e546b5dea5e4a80c407804a60940b8304335429ede3015: Status 404 returned error can't find the container with id d3837bc37806610434e546b5dea5e4a80c407804a60940b8304335429ede3015 Jul 06 19:22:57.442229 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.442208 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:22:57.450059 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:57.450032 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:57.481143 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.481110 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:57.550184 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:57.550140 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:57.654054 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:57.653953 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-103.ec2.internal\" not found" Jul 06 19:22:57.682238 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.682213 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:57.753328 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.753276 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" Jul 06 19:22:57.796367 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.796334 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:22:57.797553 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.797537 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" Jul 06 19:22:57.811781 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:57.811754 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:22:58.225421 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.225382 2573 apiserver.go:52] "Watching apiserver" Jul 06 19:22:58.233488 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.233457 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:22:58.236274 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.234967 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal","openshift-multus/multus-additional-cni-plugins-rn484","openshift-multus/multus-mgsjx","openshift-network-operator/iptables-alerter-9p9bg","openshift-ovn-kubernetes/ovnkube-node-q84g6","kube-system/konnectivity-agent-kbgkq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx","openshift-cluster-node-tuning-operator/tuned-8js94","openshift-image-registry/node-ca-qjr9z","openshift-multus/network-metrics-daemon-prgr9","openshift-network-diagnostics/network-check-target-9sjpt","kube-system/global-pull-secret-syncer-bsvw4","kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal"] Jul 06 19:22:58.237118 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.237094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.238556 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.238533 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.240720 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.240518 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.241501 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.241477 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:22:58.241879 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.241859 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:22:58.241879 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.241876 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:22:58.242171 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.242154 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l7t67\"" Jul 06 19:22:58.242697 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.242500 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.244763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.244664 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.246377 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.246290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:22:58.246973 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.246798 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:22:58.247071 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247020 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:22:58.247174 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247152 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-92rgc\"" Jul 06 19:22:58.247174 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247169 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:22:58.247373 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247203 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cf67r\"" Jul 06 19:22:58.247373 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247352 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:22:58.247483 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247391 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:22:58.247483 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247391 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:22:58.247706 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247677 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:22:58.248140 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247839 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:22:58.248140 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247879 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:22:58.248140 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.247914 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:22:58.248140 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.248106 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.248726 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.248556 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dl52g\"" Jul 06 19:22:58.248726 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.248609 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:22:58.250140 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.249707 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.250966 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.250470 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:22:58.250966 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.250643 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:22:58.250966 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.250682 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:22:58.250966 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.250819 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:22:58.250966 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.250850 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vht6l\"" Jul 06 19:22:58.251306 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.251021 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:22:58.251306 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.251060 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:22:58.252024 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.251802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:22:58.252024 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.251890 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:22:58.252187 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.252051 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:22:58.252187 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.252060 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:22:58.252327 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.252213 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:22:58.252397 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.252380 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:22:58.252557 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.252539 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:22:58.252620 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.252584 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:22:58.253693 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.253669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:22:58.253773 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.253741 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:22:58.255742 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.255705 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.255847 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.255774 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:22:58.261637 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.261601 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-76tlm\"" Jul 06 19:22:58.261637 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.261624 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5f8zd\"" Jul 06 19:22:58.261815 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.261624 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pkb6c\"" Jul 06 19:22:58.264884 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.264859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e5af044-7fed-4e7f-8037-5bf191c3a194-cni-binary-copy\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.264995 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.264896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-sysctl-conf\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.264995 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.264919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce473c59-a118-4568-99b6-c67a261353e6-host\") pod \"node-ca-qjr9z\" (UID: \"ce473c59-a118-4568-99b6-c67a261353e6\") " pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.264995 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.264949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-run-netns\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.264995 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.264974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-var-lib-cni-multus\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.265213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.264997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-var-lib-kubelet\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.265213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265019 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-host\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.265213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265048 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:22:58.265213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e221ceab-c77e-4397-8902-49677d610efb-iptables-alerter-script\") pod \"iptables-alerter-9p9bg\" (UID: \"e221ceab-c77e-4397-8902-49677d610efb\") " pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.265213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/058430a0-aad1-4846-adaf-c6f89843521c-env-overrides\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.265213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-kubernetes\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.265213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265151 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.265213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e5af044-7fed-4e7f-8037-5bf191c3a194-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.265213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e5af044-7fed-4e7f-8037-5bf191c3a194-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-daemon-config\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265284 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-system-cni-dir\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-var-lib-cni-bin\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265368 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-hostroot\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-etc-kubernetes\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-var-lib-kubelet\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-etc-openvswitch\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-lib-modules\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-sysctl-d\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce473c59-a118-4568-99b6-c67a261353e6-serviceca\") pod \"node-ca-qjr9z\" (UID: \"ce473c59-a118-4568-99b6-c67a261353e6\") " pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/058430a0-aad1-4846-adaf-c6f89843521c-ovnkube-script-lib\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265662 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-os-release\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.265696 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdf6\" (UniqueName: \"kubernetes.io/projected/b3140b06-a3b5-4734-818b-f545557c90c1-kube-api-access-zxdf6\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-os-release\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265774 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-modprobe-d\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-sysconfig\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-var-lib-openvswitch\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-node-log\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-device-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-socket-dir-parent\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-run-netns\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.265996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-run\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266024 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-run-ovn-kubernetes\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266048 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-sys-fs\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266076 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f79f1647-3ed9-4e04-8589-2b547e769cc9-konnectivity-ca\") pod \"konnectivity-agent-kbgkq\" (UID: \"f79f1647-3ed9-4e04-8589-2b547e769cc9\") " pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-tuned\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-kubelet\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bccsx\" (UniqueName: \"kubernetes.io/projected/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-kube-api-access-bccsx\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:22:58.266388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266233 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-cni-binary-copy\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c97194b-68cc-44c3-821c-893d8ee82f63-tmp\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-run-ovn\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/058430a0-aad1-4846-adaf-c6f89843521c-ovnkube-config\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/058430a0-aad1-4846-adaf-c6f89843521c-ovn-node-metrics-cert\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-cnibin\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266445 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-run-k8s-cni-cncf-io\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-cni-dir\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-systemd-units\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6ht\" (UniqueName: \"kubernetes.io/projected/058430a0-aad1-4846-adaf-c6f89843521c-kube-api-access-dr6ht\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-socket-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-cnibin\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266663 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvb5\" (UniqueName: \"kubernetes.io/projected/2e5af044-7fed-4e7f-8037-5bf191c3a194-kube-api-access-ffvb5\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-conf-dir\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-sys\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2dc\" (UniqueName: \"kubernetes.io/projected/8c97194b-68cc-44c3-821c-893d8ee82f63-kube-api-access-wp2dc\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.267162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e221ceab-c77e-4397-8902-49677d610efb-host-slash\") pod \"iptables-alerter-9p9bg\" (UID: \"e221ceab-c77e-4397-8902-49677d610efb\") " pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266807 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-registration-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-system-cni-dir\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266882 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-systemd\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwnw\" (UniqueName: \"kubernetes.io/projected/e221ceab-c77e-4397-8902-49677d610efb-kube-api-access-qzwnw\") pod \"iptables-alerter-9p9bg\" (UID: \"e221ceab-c77e-4397-8902-49677d610efb\") " pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-slash\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.266982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-run-systemd\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-cni-bin\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267031 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-cni-netd\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjjtv\" (UniqueName: \"kubernetes.io/projected/ce473c59-a118-4568-99b6-c67a261353e6-kube-api-access-vjjtv\") pod \"node-ca-qjr9z\" (UID: \"ce473c59-a118-4568-99b6-c67a261353e6\") " pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267087 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267650 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-run-multus-certs\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7rqn\" (UniqueName: \"kubernetes.io/projected/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-kube-api-access-l7rqn\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f79f1647-3ed9-4e04-8589-2b547e769cc9-agent-certs\") pod \"konnectivity-agent-kbgkq\" (UID: \"f79f1647-3ed9-4e04-8589-2b547e769cc9\") " pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-run-openvswitch\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.267977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267774 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-log-socket\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.268877 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.267797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:22:58.287352 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.287318 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:22:58.352978 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.352943 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:22:58.362498 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.362467 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lwhfb" Jul 06 19:22:58.368036 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-cni-netd\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.368174 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjtv\" (UniqueName: \"kubernetes.io/projected/ce473c59-a118-4568-99b6-c67a261353e6-kube-api-access-vjjtv\") pod \"node-ca-qjr9z\" (UID: \"ce473c59-a118-4568-99b6-c67a261353e6\") " pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.368174 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.368174 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368080 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-cni-netd\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.368174 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-run-multus-certs\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.368174 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7rqn\" (UniqueName: \"kubernetes.io/projected/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-kube-api-access-l7rqn\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.368174 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f79f1647-3ed9-4e04-8589-2b547e769cc9-agent-certs\") pod \"konnectivity-agent-kbgkq\" (UID: \"f79f1647-3ed9-4e04-8589-2b547e769cc9\") " pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-run-openvswitch\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-log-socket\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368242 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e5af044-7fed-4e7f-8037-5bf191c3a194-cni-binary-copy\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-sysctl-conf\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce473c59-a118-4568-99b6-c67a261353e6-host\") pod \"node-ca-qjr9z\" (UID: \"ce473c59-a118-4568-99b6-c67a261353e6\") " pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-run-netns\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-var-lib-cni-multus\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368397 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-var-lib-kubelet\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-host\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e221ceab-c77e-4397-8902-49677d610efb-iptables-alerter-script\") pod \"iptables-alerter-9p9bg\" (UID: \"e221ceab-c77e-4397-8902-49677d610efb\") " pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/058430a0-aad1-4846-adaf-c6f89843521c-env-overrides\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.368522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368521 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-kubernetes\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368560 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-var-lib-kubelet\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e5af044-7fed-4e7f-8037-5bf191c3a194-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e5af044-7fed-4e7f-8037-5bf191c3a194-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368615 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-sysctl-conf\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-daemon-config\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368639 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-log-socket\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368625 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368687 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-var-lib-cni-multus\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-system-cni-dir\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368726 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-run-multus-certs\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-var-lib-cni-bin\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-hostroot\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-etc-kubernetes\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-var-lib-kubelet\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.368825 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:58.369442 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-etc-openvswitch\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368616 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce473c59-a118-4568-99b6-c67a261353e6-host\") pod \"node-ca-qjr9z\" (UID: \"ce473c59-a118-4568-99b6-c67a261353e6\") " pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-lib-modules\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.368900 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs podName:7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4 nodeName:}" failed. No retries permitted until 2026-07-06 19:22:58.868876239 +0000 UTC m=+2.966103989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs") pod "network-metrics-daemon-prgr9" (UID: "7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368951 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-system-cni-dir\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368974 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-lib-modules\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368998 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-var-lib-cni-bin\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369015 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-hostroot\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-sysctl-d\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369054 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-etc-kubernetes\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce473c59-a118-4568-99b6-c67a261353e6-serviceca\") pod \"node-ca-qjr9z\" (UID: \"ce473c59-a118-4568-99b6-c67a261353e6\") " pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369110 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-var-lib-kubelet\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369128 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/058430a0-aad1-4846-adaf-c6f89843521c-ovnkube-script-lib\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-os-release\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdf6\" (UniqueName: \"kubernetes.io/projected/b3140b06-a3b5-4734-818b-f545557c90c1-kube-api-access-zxdf6\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-os-release\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-daemon-config\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.370054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369232 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-modprobe-d\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369334 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-modprobe-d\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369416 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-sysctl-d\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369419 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e5af044-7fed-4e7f-8037-5bf191c3a194-cni-binary-copy\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369468 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369475 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369155 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-etc-openvswitch\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.368651 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-run-netns\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369611 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-sysconfig\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369767 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-os-release\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369818 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-run-openvswitch\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-host\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369956 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/058430a0-aad1-4846-adaf-c6f89843521c-ovnkube-script-lib\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369957 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/058430a0-aad1-4846-adaf-c6f89843521c-env-overrides\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369975 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e5af044-7fed-4e7f-8037-5bf191c3a194-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.369334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-sysconfig\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370007 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-kubernetes\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.371218 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-var-lib-openvswitch\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-node-log\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-device-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370097 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e5af044-7fed-4e7f-8037-5bf191c3a194-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-socket-dir-parent\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370139 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-run-netns\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370163 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-node-log\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370170 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370114 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-os-release\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370172 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-socket-dir-parent\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-run\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370216 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-run-netns\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370223 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-device-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-run-ovn-kubernetes\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-run\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370266 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-var-lib-openvswitch\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-sys-fs\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f79f1647-3ed9-4e04-8589-2b547e769cc9-konnectivity-ca\") pod \"konnectivity-agent-kbgkq\" (UID: \"f79f1647-3ed9-4e04-8589-2b547e769cc9\") " pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:22:58.372213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370339 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce473c59-a118-4568-99b6-c67a261353e6-serviceca\") pod \"node-ca-qjr9z\" (UID: \"ce473c59-a118-4568-99b6-c67a261353e6\") " pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-tuned\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370327 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-run-ovn-kubernetes\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370372 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-sys-fs\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-kubelet\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bccsx\" (UniqueName: \"kubernetes.io/projected/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-kube-api-access-bccsx\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-cni-binary-copy\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c97194b-68cc-44c3-821c-893d8ee82f63-tmp\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370465 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-kubelet\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-run-ovn\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370520 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e221ceab-c77e-4397-8902-49677d610efb-iptables-alerter-script\") pod \"iptables-alerter-9p9bg\" (UID: \"e221ceab-c77e-4397-8902-49677d610efb\") " pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/058430a0-aad1-4846-adaf-c6f89843521c-ovnkube-config\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/058430a0-aad1-4846-adaf-c6f89843521c-ovn-node-metrics-cert\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370672 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-run-ovn\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370682 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-cnibin\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370742 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-cnibin\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-cni-binary-copy\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.370972 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f79f1647-3ed9-4e04-8589-2b547e769cc9-konnectivity-ca\") pod \"konnectivity-agent-kbgkq\" (UID: \"f79f1647-3ed9-4e04-8589-2b547e769cc9\") " pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:22:58.373192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-run-k8s-cni-cncf-io\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-cni-dir\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-systemd-units\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6ht\" (UniqueName: \"kubernetes.io/projected/058430a0-aad1-4846-adaf-c6f89843521c-kube-api-access-dr6ht\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371125 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-host-run-k8s-cni-cncf-io\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371141 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-kubelet-config\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-dbus\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371182 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-systemd-units\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-socket-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-cnibin\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371241 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-cni-dir\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371279 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-cnibin\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371197 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/058430a0-aad1-4846-adaf-c6f89843521c-ovnkube-config\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371355 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-socket-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvb5\" (UniqueName: \"kubernetes.io/projected/2e5af044-7fed-4e7f-8037-5bf191c3a194-kube-api-access-ffvb5\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-conf-dir\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-sys\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.374128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2dc\" (UniqueName: \"kubernetes.io/projected/8c97194b-68cc-44c3-821c-893d8ee82f63-kube-api-access-wp2dc\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e221ceab-c77e-4397-8902-49677d610efb-host-slash\") pod \"iptables-alerter-9p9bg\" (UID: \"e221ceab-c77e-4397-8902-49677d610efb\") " pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-registration-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-system-cni-dir\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-systemd\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371620 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwnw\" (UniqueName: \"kubernetes.io/projected/e221ceab-c77e-4397-8902-49677d610efb-kube-api-access-qzwnw\") pod \"iptables-alerter-9p9bg\" (UID: \"e221ceab-c77e-4397-8902-49677d610efb\") " pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-slash\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-run-systemd\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371679 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-multus-conf-dir\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-cni-bin\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371745 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-sys\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371768 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-cni-bin\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371824 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3140b06-a3b5-4734-818b-f545557c90c1-registration-dir\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371866 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e5af044-7fed-4e7f-8037-5bf191c3a194-system-cni-dir\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371915 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-systemd\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.371980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e221ceab-c77e-4397-8902-49677d610efb-host-slash\") pod \"iptables-alerter-9p9bg\" (UID: \"e221ceab-c77e-4397-8902-49677d610efb\") " pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.372108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-host-slash\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.375050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.372154 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/058430a0-aad1-4846-adaf-c6f89843521c-run-systemd\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.375930 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.373094 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f79f1647-3ed9-4e04-8589-2b547e769cc9-agent-certs\") pod \"konnectivity-agent-kbgkq\" (UID: \"f79f1647-3ed9-4e04-8589-2b547e769cc9\") " pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:22:58.375930 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.373230 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/058430a0-aad1-4846-adaf-c6f89843521c-ovn-node-metrics-cert\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.375930 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.373836 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c97194b-68cc-44c3-821c-893d8ee82f63-tmp\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.376071 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.375950 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c97194b-68cc-44c3-821c-893d8ee82f63-etc-tuned\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.387673 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.387646 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lwhfb" Jul 06 19:22:58.393305 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.393238 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" event={"ID":"ce19eea5e92a6ee78e17d806a018297b","Type":"ContainerStarted","Data":"d3837bc37806610434e546b5dea5e4a80c407804a60940b8304335429ede3015"} Jul 06 19:22:58.394660 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.394630 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" event={"ID":"76adf7ddaf5e685a8613fa423bc7c3be","Type":"ContainerStarted","Data":"56479e0793cc327995dff15c684ac4edc4b21a56ccfec266f9aa949697a1d6c3"} Jul 06 19:22:58.400666 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.400635 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2dc\" (UniqueName: \"kubernetes.io/projected/8c97194b-68cc-44c3-821c-893d8ee82f63-kube-api-access-wp2dc\") pod \"tuned-8js94\" (UID: \"8c97194b-68cc-44c3-821c-893d8ee82f63\") " pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.405439 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.405218 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6ht\" (UniqueName: \"kubernetes.io/projected/058430a0-aad1-4846-adaf-c6f89843521c-kube-api-access-dr6ht\") pod \"ovnkube-node-q84g6\" (UID: \"058430a0-aad1-4846-adaf-c6f89843521c\") " pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.405439 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.405335 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:22:58.405439 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.405355 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:22:58.405439 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.405373 2573 projected.go:196] Error preparing data for projected volume kube-api-access-f968j for pod openshift-network-diagnostics/network-check-target-9sjpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:58.405726 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.405454 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j podName:72b43de8-9bee-4860-bd8e-edc25cde27fd nodeName:}" failed. No retries permitted until 2026-07-06 19:22:58.905432306 +0000 UTC m=+3.002660052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f968j" (UniqueName: "kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j") pod "network-check-target-9sjpt" (UID: "72b43de8-9bee-4860-bd8e-edc25cde27fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:58.406868 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.406839 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwnw\" (UniqueName: \"kubernetes.io/projected/e221ceab-c77e-4397-8902-49677d610efb-kube-api-access-qzwnw\") pod \"iptables-alerter-9p9bg\" (UID: \"e221ceab-c77e-4397-8902-49677d610efb\") " pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.407735 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.407709 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjtv\" (UniqueName: \"kubernetes.io/projected/ce473c59-a118-4568-99b6-c67a261353e6-kube-api-access-vjjtv\") pod \"node-ca-qjr9z\" (UID: \"ce473c59-a118-4568-99b6-c67a261353e6\") " pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.408240 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.408215 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdf6\" (UniqueName: \"kubernetes.io/projected/b3140b06-a3b5-4734-818b-f545557c90c1-kube-api-access-zxdf6\") pod \"aws-ebs-csi-driver-node-vb2bx\" (UID: \"b3140b06-a3b5-4734-818b-f545557c90c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.408775 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.408750 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bccsx\" (UniqueName: \"kubernetes.io/projected/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-kube-api-access-bccsx\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:22:58.409801 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.409671 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvb5\" (UniqueName: \"kubernetes.io/projected/2e5af044-7fed-4e7f-8037-5bf191c3a194-kube-api-access-ffvb5\") pod \"multus-additional-cni-plugins-rn484\" (UID: \"2e5af044-7fed-4e7f-8037-5bf191c3a194\") " pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.409801 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.409778 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7rqn\" (UniqueName: \"kubernetes.io/projected/8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2-kube-api-access-l7rqn\") pod \"multus-mgsjx\" (UID: \"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2\") " pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.472307 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.472269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.472505 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.472363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-kubelet-config\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.472505 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.472388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-dbus\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.472505 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.472406 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:58.472505 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.472465 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret podName:6f20bf0b-a1f4-43f3-800d-a892bff4c7fe nodeName:}" failed. No retries permitted until 2026-07-06 19:22:58.972446338 +0000 UTC m=+3.069674088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret") pod "global-pull-secret-syncer-bsvw4" (UID: "6f20bf0b-a1f4-43f3-800d-a892bff4c7fe") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:58.472739 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.472543 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-kubelet-config\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.472739 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.472550 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-dbus\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.551579 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.551491 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" Jul 06 19:22:58.560187 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.560161 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rn484" Jul 06 19:22:58.571214 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.571168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mgsjx" Jul 06 19:22:58.577781 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.577753 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9p9bg" Jul 06 19:22:58.586983 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.586961 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:22:58.595919 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.595888 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:22:58.605825 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.605797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8js94" Jul 06 19:22:58.612708 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.612684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qjr9z" Jul 06 19:22:58.693142 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.693106 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:58.876677 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.876640 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:22:58.876837 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.876813 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:58.876904 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.876892 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs podName:7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4 nodeName:}" failed. No retries permitted until 2026-07-06 19:22:59.876869712 +0000 UTC m=+3.974097463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs") pod "network-metrics-daemon-prgr9" (UID: "7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:58.977877 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.977845 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:22:58.978055 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:58.977950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:58.978055 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.978015 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:22:58.978055 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.978035 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:22:58.978055 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.978049 2573 projected.go:196] Error preparing data for projected volume kube-api-access-f968j for pod openshift-network-diagnostics/network-check-target-9sjpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:58.978288 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.978069 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:58.978288 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.978104 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j podName:72b43de8-9bee-4860-bd8e-edc25cde27fd nodeName:}" failed. No retries permitted until 2026-07-06 19:22:59.978088145 +0000 UTC m=+4.075315880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f968j" (UniqueName: "kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j") pod "network-check-target-9sjpt" (UID: "72b43de8-9bee-4860-bd8e-edc25cde27fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:58.978288 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:58.978126 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret podName:6f20bf0b-a1f4-43f3-800d-a892bff4c7fe nodeName:}" failed. No retries permitted until 2026-07-06 19:22:59.978109847 +0000 UTC m=+4.075337601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret") pod "global-pull-secret-syncer-bsvw4" (UID: "6f20bf0b-a1f4-43f3-800d-a892bff4c7fe") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:59.389139 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:59.389103 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:22:59.389611 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:59.389119 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:22:59.389611 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.389261 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:22:59.389611 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:59.389310 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:59.389611 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.389431 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:22:59.389611 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.389501 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:22:59.391189 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:59.391155 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:17:59 +0000 UTC" deadline="2028-03-27 02:52:37.338099146 +0000 UTC" Jul 06 19:22:59.391189 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:59.391182 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15103h29m37.946920987s" Jul 06 19:22:59.636541 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:59.636511 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79f1647_3ed9_4e04_8589_2b547e769cc9.slice/crio-1befc579049a04d75255073f7d1f3bc05f922a1ba190534c86d6e0d1ace0f894 WatchSource:0}: Error finding container 1befc579049a04d75255073f7d1f3bc05f922a1ba190534c86d6e0d1ace0f894: Status 404 returned error can't find the container with id 1befc579049a04d75255073f7d1f3bc05f922a1ba190534c86d6e0d1ace0f894 Jul 06 19:22:59.638095 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:59.638068 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5af044_7fed_4e7f_8037_5bf191c3a194.slice/crio-4515bf7510a7c80cd23b78c5a6bea2c13b41a91185fd6bdebe57780c6a5065a8 WatchSource:0}: Error finding container 4515bf7510a7c80cd23b78c5a6bea2c13b41a91185fd6bdebe57780c6a5065a8: Status 404 returned error can't find the container with id 4515bf7510a7c80cd23b78c5a6bea2c13b41a91185fd6bdebe57780c6a5065a8 Jul 06 19:22:59.643634 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:59.643404 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c97194b_68cc_44c3_821c_893d8ee82f63.slice/crio-5101cf7329b774f6d8e18f0092220aa94eef07a6f6c298ba7c6d1a6dda5264d0 WatchSource:0}: Error finding container 5101cf7329b774f6d8e18f0092220aa94eef07a6f6c298ba7c6d1a6dda5264d0: Status 404 returned error can't find the container with id 5101cf7329b774f6d8e18f0092220aa94eef07a6f6c298ba7c6d1a6dda5264d0 Jul 06 19:22:59.645193 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:59.645060 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode221ceab_c77e_4397_8902_49677d610efb.slice/crio-2d4b662210d92e0157488e2a47f83812d45c2ff6cce0d6fc598c5df9e1f09b76 WatchSource:0}: Error finding container 2d4b662210d92e0157488e2a47f83812d45c2ff6cce0d6fc598c5df9e1f09b76: Status 404 returned error can't find the container with id 2d4b662210d92e0157488e2a47f83812d45c2ff6cce0d6fc598c5df9e1f09b76 Jul 06 19:22:59.645912 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:59.645814 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3140b06_a3b5_4734_818b_f545557c90c1.slice/crio-aa7cb6e75aafb51d1e9dd779a9c014fd184342dc314971b7132de5eb4ef6d867 WatchSource:0}: Error finding container aa7cb6e75aafb51d1e9dd779a9c014fd184342dc314971b7132de5eb4ef6d867: Status 404 returned error can't find the container with id aa7cb6e75aafb51d1e9dd779a9c014fd184342dc314971b7132de5eb4ef6d867 Jul 06 19:22:59.648104 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:59.647962 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce473c59_a118_4568_99b6_c67a261353e6.slice/crio-7a0223ee5ad813708048bb0804d03e8c09db5a84f3f82026e15d718eedb310ec WatchSource:0}: Error finding container 7a0223ee5ad813708048bb0804d03e8c09db5a84f3f82026e15d718eedb310ec: Status 404 returned error can't find the container with id 7a0223ee5ad813708048bb0804d03e8c09db5a84f3f82026e15d718eedb310ec Jul 06 19:22:59.649311 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:59.649156 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5a98a3_ee03_42c9_9fa8_65593aa2b8d2.slice/crio-614df986d4a559d42fb372f53b7db41668cc307331ab05553db2808cfcb8734d WatchSource:0}: Error finding container 614df986d4a559d42fb372f53b7db41668cc307331ab05553db2808cfcb8734d: Status 404 returned error can't find the container with id 614df986d4a559d42fb372f53b7db41668cc307331ab05553db2808cfcb8734d Jul 06 19:22:59.650766 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:22:59.650668 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058430a0_aad1_4846_adaf_c6f89843521c.slice/crio-ce7ee5ad46c96731d9e2ab90153812da49aaed0099ea4ffa90d263b30fb675a5 WatchSource:0}: Error finding container ce7ee5ad46c96731d9e2ab90153812da49aaed0099ea4ffa90d263b30fb675a5: Status 404 returned error can't find the container with id ce7ee5ad46c96731d9e2ab90153812da49aaed0099ea4ffa90d263b30fb675a5 Jul 06 19:22:59.884189 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:59.884158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:22:59.884350 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.884334 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:59.884409 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.884399 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs podName:7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:01.884382114 +0000 UTC m=+5.981609850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs") pod "network-metrics-daemon-prgr9" (UID: "7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:59.985040 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:59.985004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:22:59.985040 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:22:59.985046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:22:59.985268 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.985163 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:22:59.985268 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.985182 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:22:59.985268 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.985192 2573 projected.go:196] Error preparing data for projected volume kube-api-access-f968j for pod openshift-network-diagnostics/network-check-target-9sjpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:59.985268 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.985258 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j podName:72b43de8-9bee-4860-bd8e-edc25cde27fd nodeName:}" failed. No retries permitted until 2026-07-06 19:23:01.985226814 +0000 UTC m=+6.082454550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f968j" (UniqueName: "kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j") pod "network-check-target-9sjpt" (UID: "72b43de8-9bee-4860-bd8e-edc25cde27fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:59.985268 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.985164 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:59.985436 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:22:59.985298 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret podName:6f20bf0b-a1f4-43f3-800d-a892bff4c7fe nodeName:}" failed. No retries permitted until 2026-07-06 19:23:01.985289812 +0000 UTC m=+6.082517549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret") pod "global-pull-secret-syncer-bsvw4" (UID: "6f20bf0b-a1f4-43f3-800d-a892bff4c7fe") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:00.291553 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.291462 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:23:00.392328 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.392261 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:17:59 +0000 UTC" deadline="2028-02-03 23:10:31.531619743 +0000 UTC" Jul 06 19:23:00.392328 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.392301 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13851h47m31.139324815s" Jul 06 19:23:00.411298 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.411241 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" event={"ID":"ce19eea5e92a6ee78e17d806a018297b","Type":"ContainerStarted","Data":"1893d2371a017fae945ffaef9d33ee0c7e68802756e34b9438f095879c860da0"} Jul 06 19:23:00.414539 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.414480 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qjr9z" event={"ID":"ce473c59-a118-4568-99b6-c67a261353e6","Type":"ContainerStarted","Data":"7a0223ee5ad813708048bb0804d03e8c09db5a84f3f82026e15d718eedb310ec"} Jul 06 19:23:00.417115 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.417086 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mgsjx" event={"ID":"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2","Type":"ContainerStarted","Data":"614df986d4a559d42fb372f53b7db41668cc307331ab05553db2808cfcb8734d"} Jul 06 19:23:00.422050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.422014 2573 generic.go:356] "Generic (PLEG): container finished" podID="76adf7ddaf5e685a8613fa423bc7c3be" containerID="903435eb91f1d170e864cf341d64936cbc69672b4cf91c2dfdfdde8c719069a8" exitCode=0 Jul 06 19:23:00.422212 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.422140 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" event={"ID":"76adf7ddaf5e685a8613fa423bc7c3be","Type":"ContainerDied","Data":"903435eb91f1d170e864cf341d64936cbc69672b4cf91c2dfdfdde8c719069a8"} Jul 06 19:23:00.425193 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.425163 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" event={"ID":"058430a0-aad1-4846-adaf-c6f89843521c","Type":"ContainerStarted","Data":"ce7ee5ad46c96731d9e2ab90153812da49aaed0099ea4ffa90d263b30fb675a5"} Jul 06 19:23:00.429369 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.429310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" event={"ID":"b3140b06-a3b5-4734-818b-f545557c90c1","Type":"ContainerStarted","Data":"aa7cb6e75aafb51d1e9dd779a9c014fd184342dc314971b7132de5eb4ef6d867"} Jul 06 19:23:00.432263 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.432226 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9p9bg" event={"ID":"e221ceab-c77e-4397-8902-49677d610efb","Type":"ContainerStarted","Data":"2d4b662210d92e0157488e2a47f83812d45c2ff6cce0d6fc598c5df9e1f09b76"} Jul 06 19:23:00.436413 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.436320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8js94" event={"ID":"8c97194b-68cc-44c3-821c-893d8ee82f63","Type":"ContainerStarted","Data":"5101cf7329b774f6d8e18f0092220aa94eef07a6f6c298ba7c6d1a6dda5264d0"} Jul 06 19:23:00.439899 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.439826 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rn484" event={"ID":"2e5af044-7fed-4e7f-8037-5bf191c3a194","Type":"ContainerStarted","Data":"4515bf7510a7c80cd23b78c5a6bea2c13b41a91185fd6bdebe57780c6a5065a8"} Jul 06 19:23:00.456428 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.453997 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kbgkq" event={"ID":"f79f1647-3ed9-4e04-8589-2b547e769cc9","Type":"ContainerStarted","Data":"1befc579049a04d75255073f7d1f3bc05f922a1ba190534c86d6e0d1ace0f894"} Jul 06 19:23:00.470555 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:00.469193 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-103.ec2.internal" podStartSLOduration=2.46916971 podStartE2EDuration="2.46916971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:00.437028364 +0000 UTC m=+4.534256124" watchObservedRunningTime="2026-07-06 19:23:00.46916971 +0000 UTC m=+4.566397469" Jul 06 19:23:01.388342 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:01.388302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:01.388601 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:01.388451 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:01.388953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:01.388917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:01.389102 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:01.389038 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:01.389162 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:01.389102 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:01.389217 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:01.389175 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:01.466956 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:01.466878 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" event={"ID":"76adf7ddaf5e685a8613fa423bc7c3be","Type":"ContainerStarted","Data":"b5c11925c5f0a3b726c98a01b5711a5b3512287bc9012ec91f250330cac1d4e5"} Jul 06 19:23:01.902509 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:01.901883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:01.902509 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:01.902042 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:01.902509 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:01.902109 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs podName:7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:05.902088333 +0000 UTC m=+9.999316071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs") pod "network-metrics-daemon-prgr9" (UID: "7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:02.003647 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:02.002761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:02.003647 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:02.002822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:02.003647 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:02.002986 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:02.003647 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:02.003010 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:02.003647 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:02.003024 2573 projected.go:196] Error preparing data for projected volume kube-api-access-f968j for pod openshift-network-diagnostics/network-check-target-9sjpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:02.003647 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:02.003095 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j podName:72b43de8-9bee-4860-bd8e-edc25cde27fd nodeName:}" failed. No retries permitted until 2026-07-06 19:23:06.003070651 +0000 UTC m=+10.100298387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f968j" (UniqueName: "kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j") pod "network-check-target-9sjpt" (UID: "72b43de8-9bee-4860-bd8e-edc25cde27fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:02.003647 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:02.003553 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:02.003647 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:02.003604 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret podName:6f20bf0b-a1f4-43f3-800d-a892bff4c7fe nodeName:}" failed. No retries permitted until 2026-07-06 19:23:06.003588529 +0000 UTC m=+10.100816269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret") pod "global-pull-secret-syncer-bsvw4" (UID: "6f20bf0b-a1f4-43f3-800d-a892bff4c7fe") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:03.388553 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:03.388513 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:03.389059 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:03.388570 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:03.389059 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:03.388535 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:03.389059 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:03.388677 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:03.389059 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:03.388787 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:03.389059 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:03.388876 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:05.388412 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:05.388373 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:05.388412 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:05.388411 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:05.389023 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:05.388416 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:05.389023 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:05.388516 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:05.389023 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:05.388640 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:05.389023 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:05.388741 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:05.937526 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:05.937488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:05.937746 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:05.937715 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:05.937805 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:05.937789 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs podName:7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:13.937772393 +0000 UTC m=+18.035000129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs") pod "network-metrics-daemon-prgr9" (UID: "7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:06.038607 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:06.038248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:06.038607 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:06.038313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:06.038607 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:06.038442 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:06.038607 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:06.038486 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:06.038607 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:06.038504 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:06.038607 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:06.038518 2573 projected.go:196] Error preparing data for projected volume kube-api-access-f968j for pod openshift-network-diagnostics/network-check-target-9sjpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:06.038607 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:06.038529 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret podName:6f20bf0b-a1f4-43f3-800d-a892bff4c7fe nodeName:}" failed. No retries permitted until 2026-07-06 19:23:14.038507212 +0000 UTC m=+18.135734952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret") pod "global-pull-secret-syncer-bsvw4" (UID: "6f20bf0b-a1f4-43f3-800d-a892bff4c7fe") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:06.038607 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:06.038570 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j podName:72b43de8-9bee-4860-bd8e-edc25cde27fd nodeName:}" failed. No retries permitted until 2026-07-06 19:23:14.038557358 +0000 UTC m=+18.135785092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f968j" (UniqueName: "kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j") pod "network-check-target-9sjpt" (UID: "72b43de8-9bee-4860-bd8e-edc25cde27fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:07.388686 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:07.388156 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:07.388686 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:07.388185 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:07.388686 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:07.388185 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:07.388686 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:07.388329 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:07.389414 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:07.388768 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:07.389414 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:07.388863 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:09.389143 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:09.388460 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:09.389143 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:09.388503 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:09.389143 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:09.388503 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:09.389143 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:09.388627 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:09.389143 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:09.389019 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:09.389143 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:09.389109 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:10.486419 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:10.486374 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rn484" event={"ID":"2e5af044-7fed-4e7f-8037-5bf191c3a194","Type":"ContainerStarted","Data":"ec6ce5934eb2551c2164b393decb359209d19c7bf2488660a2452bfa89199add"} Jul 06 19:23:10.488294 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:10.488264 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kbgkq" event={"ID":"f79f1647-3ed9-4e04-8589-2b547e769cc9","Type":"ContainerStarted","Data":"977cb0ed993514140816fffc01aa1190bfae76eab374dd1a72b8689e4fd32503"} Jul 06 19:23:10.490506 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:10.490479 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qjr9z" event={"ID":"ce473c59-a118-4568-99b6-c67a261353e6","Type":"ContainerStarted","Data":"fcf04c189d72f8b25ef2d8f7032e7f9f512c39a5dfa2da2a20cb743a730740de"} Jul 06 19:23:10.492329 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:10.492273 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" event={"ID":"b3140b06-a3b5-4734-818b-f545557c90c1","Type":"ContainerStarted","Data":"f1c114d34441add247debcaa2b9cf042e3022675a302f6010cb92fab686b7702"} Jul 06 19:23:10.512155 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:10.512103 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-103.ec2.internal" podStartSLOduration=12.512058884 podStartE2EDuration="12.512058884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:01.519065281 +0000 UTC m=+5.616293040" watchObservedRunningTime="2026-07-06 19:23:10.512058884 +0000 UTC m=+14.609286634" Jul 06 19:23:10.554974 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:10.554912 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kbgkq" podStartSLOduration=2.962109006 podStartE2EDuration="13.554891302s" totalImagesPullingTime="10.592782296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:10.533265072 +0000 UTC m=+14.630492828" watchObservedRunningTime="2026-07-06 19:23:10.554891302 +0000 UTC m=+14.652119058" Jul 06 19:23:10.555261 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:10.555199 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qjr9z" podStartSLOduration=2.991987882 podStartE2EDuration="13.555191617s" totalImagesPullingTime="10.563203735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:10.554378765 +0000 UTC m=+14.651606533" watchObservedRunningTime="2026-07-06 19:23:10.555191617 +0000 UTC m=+14.652419375" Jul 06 19:23:11.388556 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.388516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:11.388556 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.388548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:11.388803 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.388516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:11.388803 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:11.388642 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:11.388803 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:11.388715 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:11.388939 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:11.388811 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:11.556008 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.555971 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:11.588937 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.588904 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:23:11.590014 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.589993 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:23:11.685945 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.685871 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:11.774699 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.774669 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-gdpq6"] Jul 06 19:23:11.794144 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.794120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:11.796999 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.796976 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8tm5l\"" Jul 06 19:23:11.797144 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.797094 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:23:11.797144 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.797120 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:23:11.837051 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.836974 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:11.886432 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.886393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6b7w\" (UniqueName: \"kubernetes.io/projected/e0eeb8e3-8400-49e8-89cb-3977de6b1b6d-kube-api-access-s6b7w\") pod \"node-resolver-gdpq6\" (UID: \"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d\") " pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:11.886577 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.886451 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0eeb8e3-8400-49e8-89cb-3977de6b1b6d-tmp-dir\") pod \"node-resolver-gdpq6\" (UID: \"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d\") " pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:11.886577 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.886547 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0eeb8e3-8400-49e8-89cb-3977de6b1b6d-hosts-file\") pod \"node-resolver-gdpq6\" (UID: \"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d\") " pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:11.918134 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.918089 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:11.987798 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.987715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6b7w\" (UniqueName: \"kubernetes.io/projected/e0eeb8e3-8400-49e8-89cb-3977de6b1b6d-kube-api-access-s6b7w\") pod \"node-resolver-gdpq6\" (UID: \"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d\") " pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:11.987798 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.987781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0eeb8e3-8400-49e8-89cb-3977de6b1b6d-tmp-dir\") pod \"node-resolver-gdpq6\" (UID: \"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d\") " pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:11.987998 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.987816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0eeb8e3-8400-49e8-89cb-3977de6b1b6d-hosts-file\") pod \"node-resolver-gdpq6\" (UID: \"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d\") " pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:11.987998 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.987915 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0eeb8e3-8400-49e8-89cb-3977de6b1b6d-hosts-file\") pod \"node-resolver-gdpq6\" (UID: \"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d\") " pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:11.988138 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.988123 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0eeb8e3-8400-49e8-89cb-3977de6b1b6d-tmp-dir\") pod \"node-resolver-gdpq6\" (UID: \"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d\") " pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:11.999800 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:11.999769 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6b7w\" (UniqueName: \"kubernetes.io/projected/e0eeb8e3-8400-49e8-89cb-3977de6b1b6d-kube-api-access-s6b7w\") pod \"node-resolver-gdpq6\" (UID: \"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d\") " pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:12.105190 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:12.105149 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gdpq6" Jul 06 19:23:12.496275 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:12.496238 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:23:12.496731 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:12.496714 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kbgkq" Jul 06 19:23:12.590519 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:12.590484 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:12.777421 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:12.777327 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:12.875971 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:12.875941 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:12.975822 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:12.975786 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:13.388454 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:13.388420 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:13.388454 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:13.388455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:13.388674 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:13.388455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:13.388674 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:13.388558 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:13.388780 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:13.388672 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:13.388833 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:13.388789 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:13.999668 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:13.999630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:14.000168 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:13.999743 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:14.000168 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:13.999810 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs podName:7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:29.999791293 +0000 UTC m=+34.097019028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs") pod "network-metrics-daemon-prgr9" (UID: "7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:14.100914 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:14.100871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:14.101074 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:14.100926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:14.101074 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:14.101019 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:14.101074 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:14.101059 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:14.101074 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:14.101076 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:14.101235 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:14.101089 2573 projected.go:196] Error preparing data for projected volume kube-api-access-f968j for pod openshift-network-diagnostics/network-check-target-9sjpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:14.101235 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:14.101094 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret podName:6f20bf0b-a1f4-43f3-800d-a892bff4c7fe nodeName:}" failed. No retries permitted until 2026-07-06 19:23:30.101079277 +0000 UTC m=+34.198307010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret") pod "global-pull-secret-syncer-bsvw4" (UID: "6f20bf0b-a1f4-43f3-800d-a892bff4c7fe") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:14.101235 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:14.101138 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j podName:72b43de8-9bee-4860-bd8e-edc25cde27fd nodeName:}" failed. No retries permitted until 2026-07-06 19:23:30.101121492 +0000 UTC m=+34.198349227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f968j" (UniqueName: "kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j") pod "network-check-target-9sjpt" (UID: "72b43de8-9bee-4860-bd8e-edc25cde27fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:14.509012 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:14.508982 2573 generic.go:356] "Generic (PLEG): container finished" podID="2e5af044-7fed-4e7f-8037-5bf191c3a194" containerID="ec6ce5934eb2551c2164b393decb359209d19c7bf2488660a2452bfa89199add" exitCode=0 Jul 06 19:23:14.509194 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:14.509061 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rn484" event={"ID":"2e5af044-7fed-4e7f-8037-5bf191c3a194","Type":"ContainerDied","Data":"ec6ce5934eb2551c2164b393decb359209d19c7bf2488660a2452bfa89199add"} Jul 06 19:23:15.388317 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:15.388286 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:15.388779 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:15.388286 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:15.388779 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:15.388422 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:15.388779 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:15.388286 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:15.388779 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:15.388477 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:15.388779 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:15.388584 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:16.014545 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:16.014505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:16.118729 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:16.118684 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:16.240649 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:16.240612 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:16.311385 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:16.311300 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:16.391800 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:16.391769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:17.389006 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:17.388975 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:17.389006 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:17.389000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:17.389275 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:17.388975 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:17.389275 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:17.389129 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:17.389275 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:17.389236 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:17.389411 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:17.389289 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:18.654345 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:23:18.654311 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0eeb8e3_8400_49e8_89cb_3977de6b1b6d.slice/crio-533abc1aa55398dc4cb598239c4b69babcdcf43116b061554165043da050930c WatchSource:0}: Error finding container 533abc1aa55398dc4cb598239c4b69babcdcf43116b061554165043da050930c: Status 404 returned error can't find the container with id 533abc1aa55398dc4cb598239c4b69babcdcf43116b061554165043da050930c Jul 06 19:23:19.388920 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.388737 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:19.389044 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.388785 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:19.389044 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:19.389015 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:19.389044 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.388792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:19.389391 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:19.389112 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:19.389391 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:19.389197 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:19.436265 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.436231 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:23:19.521074 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.521037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8js94" event={"ID":"8c97194b-68cc-44c3-821c-893d8ee82f63","Type":"ContainerStarted","Data":"d3c26ba1e9b4024c553d66a30c4087d9f73e0faa22da6f80c6feabfd9ddc234f"} Jul 06 19:23:19.522554 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.522530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gdpq6" event={"ID":"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d","Type":"ContainerStarted","Data":"f2a3b8b24b073433d7872f523545d5ae7e20f27112fb5e97850fef18a03ceb48"} Jul 06 19:23:19.522661 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.522574 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gdpq6" event={"ID":"e0eeb8e3-8400-49e8-89cb-3977de6b1b6d","Type":"ContainerStarted","Data":"533abc1aa55398dc4cb598239c4b69babcdcf43116b061554165043da050930c"} Jul 06 19:23:19.524000 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.523972 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mgsjx" event={"ID":"8b5a98a3-ee03-42c9-9fa8-65593aa2b8d2","Type":"ContainerStarted","Data":"0681b9052727bb31643e25d21f02e3c0f67a2f8a561fe9d2428193b1409188b8"} Jul 06 19:23:19.526157 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.526133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" event={"ID":"058430a0-aad1-4846-adaf-c6f89843521c","Type":"ContainerStarted","Data":"6f73f56b8b2aa64a3c7e03a54d99d23d765d2d01d62a8e39455521ef8ebb070f"} Jul 06 19:23:19.526266 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.526241 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" event={"ID":"058430a0-aad1-4846-adaf-c6f89843521c","Type":"ContainerStarted","Data":"f0d5956de5fe31bbb1cd6e52e2926e3de2ccc86ba3567e2a12a5e8fba738291f"} Jul 06 19:23:19.526318 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.526295 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" event={"ID":"058430a0-aad1-4846-adaf-c6f89843521c","Type":"ContainerStarted","Data":"e064a205294a2e75e48a463cf3a08404e6c449d20e42bb7b84724723eaf6008e"} Jul 06 19:23:19.527624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.527605 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" event={"ID":"b3140b06-a3b5-4734-818b-f545557c90c1","Type":"ContainerStarted","Data":"57d3a7c62d5c9cc185d8af3036f1f038432ca1d7aebf97c85e1cfb417287aace"} Jul 06 19:23:19.541958 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.541914 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8js94" podStartSLOduration=3.536274777 podStartE2EDuration="22.541891956s" totalImagesPullingTime="19.005617179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:19.54138046 +0000 UTC m=+23.638608230" watchObservedRunningTime="2026-07-06 19:23:19.541891956 +0000 UTC m=+23.639119711" Jul 06 19:23:19.556167 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.556119 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gdpq6" podStartSLOduration=8.55610557 podStartE2EDuration="8.55610557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:19.555691626 +0000 UTC m=+23.652919381" watchObservedRunningTime="2026-07-06 19:23:19.55610557 +0000 UTC m=+23.653333325" Jul 06 19:23:19.571702 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.571661 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mgsjx" podStartSLOduration=3.570196383 podStartE2EDuration="22.571647644s" totalImagesPullingTime="19.001451261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:19.571608833 +0000 UTC m=+23.668836588" watchObservedRunningTime="2026-07-06 19:23:19.571647644 +0000 UTC m=+23.668875399" Jul 06 19:23:19.731393 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.731236 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:19.830918 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.830881 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:19.902260 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:19.902226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:20.323708 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.323582 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:23:19.436260511Z","UUID":"00e2687d-88a9-4387-a939-a51249342a94","Handler":null,"Name":"","Endpoint":""} Jul 06 19:23:20.325314 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.325290 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:23:20.325443 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.325334 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:23:20.532818 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.532733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" event={"ID":"058430a0-aad1-4846-adaf-c6f89843521c","Type":"ContainerStarted","Data":"fc8341c474da47505e51db5fd63b8e306be5474fce52e3e8c852257ba6a1a692"} Jul 06 19:23:20.533056 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.532826 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" event={"ID":"058430a0-aad1-4846-adaf-c6f89843521c","Type":"ContainerStarted","Data":"c4eb3181a0f80f3feff3e5b009e2b5e4a08495494a5a41320373036de3b78fc2"} Jul 06 19:23:20.533056 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.532871 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" event={"ID":"058430a0-aad1-4846-adaf-c6f89843521c","Type":"ContainerStarted","Data":"9129a0bd12b339fad9233e12e79331cac7e202facd0dbffe989681c057b5c9da"} Jul 06 19:23:20.534035 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.534010 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9p9bg" event={"ID":"e221ceab-c77e-4397-8902-49677d610efb","Type":"ContainerStarted","Data":"46534e458411b1c6072042782098d832cb7137f373f36871a3e009ba542924d1"} Jul 06 19:23:20.553018 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.552972 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9p9bg" podStartSLOduration=12.963741091 podStartE2EDuration="23.552959038s" totalImagesPullingTime="10.589217947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:20.552602697 +0000 UTC m=+24.649830452" watchObservedRunningTime="2026-07-06 19:23:20.552959038 +0000 UTC m=+24.650186790" Jul 06 19:23:20.698949 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.698920 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:20.887047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:20.887012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" limit=1024 Jul 06 19:23:21.388147 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:21.388106 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:21.388373 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:21.388106 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:21.388373 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:21.388251 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:21.388373 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:21.388106 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:21.388373 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:21.388328 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:21.388521 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:21.388390 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:21.538653 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:21.538608 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" event={"ID":"b3140b06-a3b5-4734-818b-f545557c90c1","Type":"ContainerStarted","Data":"ecbc4603167fed6d95a0ba61acb786144f751ae1a1917380d9d0b612261ec14e"} Jul 06 19:23:21.558824 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:21.558781 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vb2bx" podStartSLOduration=3.962852681 podStartE2EDuration="24.55876685s" totalImagesPullingTime="20.595914169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:21.558164372 +0000 UTC m=+25.655392126" watchObservedRunningTime="2026-07-06 19:23:21.55876685 +0000 UTC m=+25.655994605" Jul 06 19:23:22.543954 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:22.543916 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" event={"ID":"058430a0-aad1-4846-adaf-c6f89843521c","Type":"ContainerStarted","Data":"53cfdc2f7cf50206a55fdd8f1a5dfe9087a616f3227911c00bdc80339792b19a"} Jul 06 19:23:23.389106 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:23.389069 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:23.389317 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:23.388809 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:23.389317 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:23.389262 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:23.389317 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:23.389292 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:23.389505 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:23.389483 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:23.389819 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:23.389626 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:24.553063 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:24.552828 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" event={"ID":"058430a0-aad1-4846-adaf-c6f89843521c","Type":"ContainerStarted","Data":"74fb4eeda5b5a5246f30afc069d9651dbc3f0407c998fe97a4a2ed5ababef04e"} Jul 06 19:23:24.553063 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:24.553121 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:23:24.570815 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:24.570784 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:23:24.600273 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:24.600192 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" podStartSLOduration=8.349640128 podStartE2EDuration="27.600172375s" totalImagesPullingTime="19.250532247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:24.599939322 +0000 UTC m=+28.697167100" watchObservedRunningTime="2026-07-06 19:23:24.600172375 +0000 UTC m=+28.697400132" Jul 06 19:23:24.929538 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:24.929515 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gdpq6_e0eeb8e3-8400-49e8-89cb-3977de6b1b6d/dns-node-resolver/0.log" Jul 06 19:23:25.389078 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:25.389045 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:25.389264 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:25.389087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:25.389264 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:25.389137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:25.389264 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:25.389163 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:25.389264 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:25.389254 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:25.389467 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:25.389330 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:25.556148 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:25.556118 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:23:25.556622 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:25.556226 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:23:25.572965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:25.572937 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:23:26.324220 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:26.323978 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qjr9z_ce473c59-a118-4568-99b6-c67a261353e6/node-ca/0.log" Jul 06 19:23:26.560120 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:26.560085 2573 generic.go:356] "Generic (PLEG): container finished" podID="2e5af044-7fed-4e7f-8037-5bf191c3a194" containerID="a36f7c88b286ce254df3c20ca47b790e12a41da1cc369138b0bad5c46d758dc7" exitCode=0 Jul 06 19:23:26.560592 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:26.560157 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rn484" event={"ID":"2e5af044-7fed-4e7f-8037-5bf191c3a194","Type":"ContainerDied","Data":"a36f7c88b286ce254df3c20ca47b790e12a41da1cc369138b0bad5c46d758dc7"} Jul 06 19:23:26.566086 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:26.566066 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-prgr9"] Jul 06 19:23:26.566184 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:26.566170 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:26.566327 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:26.566303 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:26.569569 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:26.569553 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9sjpt"] Jul 06 19:23:26.569730 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:26.569633 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:26.569730 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:26.569705 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:26.570222 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:26.570184 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bsvw4"] Jul 06 19:23:26.570315 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:26.570282 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:26.570377 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:26.570346 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:27.564128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:27.564092 2573 generic.go:356] "Generic (PLEG): container finished" podID="2e5af044-7fed-4e7f-8037-5bf191c3a194" containerID="cbfdb3dfd4fd334d13b8cb2033f5512f08f4535dd73959ac3e854a630357e23c" exitCode=0 Jul 06 19:23:27.564635 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:27.564167 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rn484" event={"ID":"2e5af044-7fed-4e7f-8037-5bf191c3a194","Type":"ContainerDied","Data":"cbfdb3dfd4fd334d13b8cb2033f5512f08f4535dd73959ac3e854a630357e23c"} Jul 06 19:23:28.389112 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:28.389084 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:28.389112 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:28.389106 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:28.389317 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:28.389094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:28.389317 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:28.389243 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:28.389416 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:28.389324 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:28.389466 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:28.389441 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:28.569016 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:28.568974 2573 generic.go:356] "Generic (PLEG): container finished" podID="2e5af044-7fed-4e7f-8037-5bf191c3a194" containerID="bb791df7e79b431cfad3fd4fbeacc5e657af66b76a442fe66f52c5969f85ab0a" exitCode=0 Jul 06 19:23:28.569016 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:28.569016 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rn484" event={"ID":"2e5af044-7fed-4e7f-8037-5bf191c3a194","Type":"ContainerDied","Data":"bb791df7e79b431cfad3fd4fbeacc5e657af66b76a442fe66f52c5969f85ab0a"} Jul 06 19:23:30.016174 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:30.016135 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:30.016598 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.016289 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:30.016598 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.016345 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs podName:7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:02.016331333 +0000 UTC m=+66.113559067 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs") pod "network-metrics-daemon-prgr9" (UID: "7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:30.117017 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:30.116976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:30.117194 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:30.117029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:30.117194 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.117121 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:30.117194 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.117161 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:30.117194 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.117177 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:30.117194 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.117189 2573 projected.go:196] Error preparing data for projected volume kube-api-access-f968j for pod openshift-network-diagnostics/network-check-target-9sjpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:30.117194 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.117193 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret podName:6f20bf0b-a1f4-43f3-800d-a892bff4c7fe nodeName:}" failed. No retries permitted until 2026-07-06 19:24:02.117175318 +0000 UTC m=+66.214403066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret") pod "global-pull-secret-syncer-bsvw4" (UID: "6f20bf0b-a1f4-43f3-800d-a892bff4c7fe") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:30.117443 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.117243 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j podName:72b43de8-9bee-4860-bd8e-edc25cde27fd nodeName:}" failed. No retries permitted until 2026-07-06 19:24:02.117233345 +0000 UTC m=+66.214461079 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f968j" (UniqueName: "kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j") pod "network-check-target-9sjpt" (UID: "72b43de8-9bee-4860-bd8e-edc25cde27fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:30.388659 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:30.388621 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:30.388822 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:30.388621 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:30.388822 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.388735 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:30.388900 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.388824 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:30.388900 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:30.388630 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:30.388983 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:30.388934 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:32.389263 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:32.389058 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:32.389671 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:32.389065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:32.389671 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:32.389345 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9sjpt" podUID="72b43de8-9bee-4860-bd8e-edc25cde27fd" Jul 06 19:23:32.389671 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:32.389065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:32.389671 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:32.389470 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bsvw4" podUID="6f20bf0b-a1f4-43f3-800d-a892bff4c7fe" Jul 06 19:23:32.389671 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:23:32.389528 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prgr9" podUID="7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4" Jul 06 19:23:32.696148 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:32.696074 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-103.ec2.internal" event="NodeReady" Jul 06 19:23:32.696310 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:32.696219 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:23:34.389004 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:34.388925 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:23:34.389512 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:34.388925 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:23:34.389512 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:34.389082 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:23:34.393741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:34.393710 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:23:34.393892 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:34.393752 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:23:34.393956 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:34.393945 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:23:34.395235 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:34.394627 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tr4sh\"" Jul 06 19:23:34.395235 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:34.394719 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d9llp\"" Jul 06 19:23:34.395235 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:34.394832 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:23:35.590854 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:35.590822 2573 generic.go:356] "Generic (PLEG): container finished" podID="2e5af044-7fed-4e7f-8037-5bf191c3a194" containerID="1affa36f6d45038866f6ffecd566c758ec0158f2d2ec7ec71989c9de00dd6f48" exitCode=0 Jul 06 19:23:35.591227 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:35.590866 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rn484" event={"ID":"2e5af044-7fed-4e7f-8037-5bf191c3a194","Type":"ContainerDied","Data":"1affa36f6d45038866f6ffecd566c758ec0158f2d2ec7ec71989c9de00dd6f48"} Jul 06 19:23:36.595540 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:36.595508 2573 generic.go:356] "Generic (PLEG): container finished" podID="2e5af044-7fed-4e7f-8037-5bf191c3a194" containerID="6602187d65333e2d6e2a947f7f291271e70dfb7df8e3af039b6950015d85bd65" exitCode=0 Jul 06 19:23:36.595959 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:36.595553 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rn484" event={"ID":"2e5af044-7fed-4e7f-8037-5bf191c3a194","Type":"ContainerDied","Data":"6602187d65333e2d6e2a947f7f291271e70dfb7df8e3af039b6950015d85bd65"} Jul 06 19:23:37.603224 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:37.603173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rn484" event={"ID":"2e5af044-7fed-4e7f-8037-5bf191c3a194","Type":"ContainerStarted","Data":"91597561729d69fdf340a17eee1f82d152267dd4450a4a94c037d77247d8d45a"} Jul 06 19:23:51.440575 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.440514 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rn484" podStartSLOduration=24.460477989 podStartE2EDuration="54.440498117s" totalImagesPullingTime="29.980020128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:37.634517811 +0000 UTC m=+41.731745565" watchObservedRunningTime="2026-07-06 19:23:51.440498117 +0000 UTC m=+55.537725849" Jul 06 19:23:51.441190 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.440810 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f59cc6f6d-bnxlq"] Jul 06 19:23:51.444251 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.444227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.447543 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.447521 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:23:51.447876 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.447859 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:23:51.447993 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.447973 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:23:51.448041 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.447994 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jf8st\"" Jul 06 19:23:51.453471 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.453446 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:23:51.477735 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.477698 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f59cc6f6d-bnxlq"] Jul 06 19:23:51.483569 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.483548 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v85qf"] Jul 06 19:23:51.485686 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.485672 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sx4t5"] Jul 06 19:23:51.485808 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.485790 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v85qf" Jul 06 19:23:51.487580 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.487560 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.492990 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.492972 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qx2jx\"" Jul 06 19:23:51.493094 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.493051 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:23:51.493094 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.493075 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:23:51.493198 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.493185 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bhtlv\"" Jul 06 19:23:51.494069 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.494056 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:23:51.495737 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.495721 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:23:51.495786 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.495743 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:23:51.508506 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.508484 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v85qf"] Jul 06 19:23:51.517908 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.517883 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sx4t5"] Jul 06 19:23:51.569119 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.569088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65362076-a446-4576-ad83-6a994035a453-trusted-ca\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.569119 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.569126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65362076-a446-4576-ad83-6a994035a453-registry-tls\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.569362 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.569147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65362076-a446-4576-ad83-6a994035a453-installation-pull-secrets\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.569362 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.569172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65362076-a446-4576-ad83-6a994035a453-bound-sa-token\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.569362 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.569245 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65362076-a446-4576-ad83-6a994035a453-ca-trust-extracted\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.569362 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.569283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65362076-a446-4576-ad83-6a994035a453-image-registry-private-configuration\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.569362 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.569303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65362076-a446-4576-ad83-6a994035a453-registry-certificates\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.569362 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.569348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjp6\" (UniqueName: \"kubernetes.io/projected/65362076-a446-4576-ad83-6a994035a453-kube-api-access-2qjp6\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.574667 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.574643 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-czqtk"] Jul 06 19:23:51.576681 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.576666 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.580255 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.580223 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lq9xg\"" Jul 06 19:23:51.580255 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.580245 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:23:51.580424 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.580283 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:23:51.580424 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.580307 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:23:51.580529 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.580448 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:23:51.602123 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.602098 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-czqtk"] Jul 06 19:23:51.670302 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjp6\" (UniqueName: \"kubernetes.io/projected/65362076-a446-4576-ad83-6a994035a453-kube-api-access-2qjp6\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.670302 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79911034-6ae1-42d7-9b8a-4f370e0b3099-config-volume\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.670570 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65362076-a446-4576-ad83-6a994035a453-trusted-ca\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.670570 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65362076-a446-4576-ad83-6a994035a453-registry-tls\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.670570 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65362076-a446-4576-ad83-6a994035a453-installation-pull-secrets\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.670570 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65362076-a446-4576-ad83-6a994035a453-bound-sa-token\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.670570 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670416 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bd697ac-eb1a-4cda-a72a-6024967509f4-cert\") pod \"ingress-canary-v85qf\" (UID: \"8bd697ac-eb1a-4cda-a72a-6024967509f4\") " pod="openshift-ingress-canary/ingress-canary-v85qf" Jul 06 19:23:51.670570 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxhk\" (UniqueName: \"kubernetes.io/projected/8bd697ac-eb1a-4cda-a72a-6024967509f4-kube-api-access-gtxhk\") pod \"ingress-canary-v85qf\" (UID: \"8bd697ac-eb1a-4cda-a72a-6024967509f4\") " pod="openshift-ingress-canary/ingress-canary-v85qf" Jul 06 19:23:51.670570 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65362076-a446-4576-ad83-6a994035a453-ca-trust-extracted\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.670899 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79911034-6ae1-42d7-9b8a-4f370e0b3099-tmp-dir\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.670899 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65362076-a446-4576-ad83-6a994035a453-image-registry-private-configuration\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.670899 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670664 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79911034-6ae1-42d7-9b8a-4f370e0b3099-metrics-tls\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.670899 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65362076-a446-4576-ad83-6a994035a453-registry-certificates\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.670899 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.670729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgtlt\" (UniqueName: \"kubernetes.io/projected/79911034-6ae1-42d7-9b8a-4f370e0b3099-kube-api-access-sgtlt\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.671133 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.671071 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65362076-a446-4576-ad83-6a994035a453-ca-trust-extracted\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.671398 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.671374 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65362076-a446-4576-ad83-6a994035a453-registry-certificates\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.671577 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.671552 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65362076-a446-4576-ad83-6a994035a453-trusted-ca\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.674601 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.674577 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65362076-a446-4576-ad83-6a994035a453-installation-pull-secrets\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.674601 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.674588 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65362076-a446-4576-ad83-6a994035a453-image-registry-private-configuration\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.674756 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.674592 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65362076-a446-4576-ad83-6a994035a453-registry-tls\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.682497 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.682478 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjp6\" (UniqueName: \"kubernetes.io/projected/65362076-a446-4576-ad83-6a994035a453-kube-api-access-2qjp6\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.685858 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.685828 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65362076-a446-4576-ad83-6a994035a453-bound-sa-token\") pod \"image-registry-6f59cc6f6d-bnxlq\" (UID: \"65362076-a446-4576-ad83-6a994035a453\") " pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.754423 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.754326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:51.771494 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bd697ac-eb1a-4cda-a72a-6024967509f4-cert\") pod \"ingress-canary-v85qf\" (UID: \"8bd697ac-eb1a-4cda-a72a-6024967509f4\") " pod="openshift-ingress-canary/ingress-canary-v85qf" Jul 06 19:23:51.771603 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxhk\" (UniqueName: \"kubernetes.io/projected/8bd697ac-eb1a-4cda-a72a-6024967509f4-kube-api-access-gtxhk\") pod \"ingress-canary-v85qf\" (UID: \"8bd697ac-eb1a-4cda-a72a-6024967509f4\") " pod="openshift-ingress-canary/ingress-canary-v85qf" Jul 06 19:23:51.771603 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79911034-6ae1-42d7-9b8a-4f370e0b3099-tmp-dir\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.771603 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-crio-socket\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.771716 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.771716 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771652 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2ct\" (UniqueName: \"kubernetes.io/projected/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-kube-api-access-vs2ct\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.771716 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-data-volume\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.771820 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79911034-6ae1-42d7-9b8a-4f370e0b3099-metrics-tls\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.771820 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgtlt\" (UniqueName: \"kubernetes.io/projected/79911034-6ae1-42d7-9b8a-4f370e0b3099-kube-api-access-sgtlt\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.771948 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771892 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79911034-6ae1-42d7-9b8a-4f370e0b3099-tmp-dir\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.772025 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.771975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79911034-6ae1-42d7-9b8a-4f370e0b3099-config-volume\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.772025 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.772013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.772448 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.772430 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79911034-6ae1-42d7-9b8a-4f370e0b3099-config-volume\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.773813 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.773796 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bd697ac-eb1a-4cda-a72a-6024967509f4-cert\") pod \"ingress-canary-v85qf\" (UID: \"8bd697ac-eb1a-4cda-a72a-6024967509f4\") " pod="openshift-ingress-canary/ingress-canary-v85qf" Jul 06 19:23:51.773898 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.773860 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79911034-6ae1-42d7-9b8a-4f370e0b3099-metrics-tls\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.830031 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.830002 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgtlt\" (UniqueName: \"kubernetes.io/projected/79911034-6ae1-42d7-9b8a-4f370e0b3099-kube-api-access-sgtlt\") pod \"dns-default-sx4t5\" (UID: \"79911034-6ae1-42d7-9b8a-4f370e0b3099\") " pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:51.831438 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.831415 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxhk\" (UniqueName: \"kubernetes.io/projected/8bd697ac-eb1a-4cda-a72a-6024967509f4-kube-api-access-gtxhk\") pod \"ingress-canary-v85qf\" (UID: \"8bd697ac-eb1a-4cda-a72a-6024967509f4\") " pod="openshift-ingress-canary/ingress-canary-v85qf" Jul 06 19:23:51.872884 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.872771 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.872884 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.872870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-crio-socket\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.873074 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.872911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.873074 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.872937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2ct\" (UniqueName: \"kubernetes.io/projected/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-kube-api-access-vs2ct\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.873074 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.872967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-data-volume\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.873367 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.873255 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-data-volume\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.873367 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.873337 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-crio-socket\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.873538 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.873383 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.875366 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.875337 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.894039 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.894014 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2ct\" (UniqueName: \"kubernetes.io/projected/c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4-kube-api-access-vs2ct\") pod \"insights-runtime-extractor-czqtk\" (UID: \"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4\") " pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:51.999765 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:51.999718 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f59cc6f6d-bnxlq"] Jul 06 19:23:52.003106 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:23:52.003076 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65362076_a446_4576_ad83_6a994035a453.slice/crio-83f1f4547b7a106740cb87a2343249609ced525792efcfc54f461322a4ab8dd3 WatchSource:0}: Error finding container 83f1f4547b7a106740cb87a2343249609ced525792efcfc54f461322a4ab8dd3: Status 404 returned error can't find the container with id 83f1f4547b7a106740cb87a2343249609ced525792efcfc54f461322a4ab8dd3 Jul 06 19:23:52.095165 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.095131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v85qf" Jul 06 19:23:52.099954 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.099938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:52.185611 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.185297 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-czqtk" Jul 06 19:23:52.235664 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.235637 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v85qf"] Jul 06 19:23:52.237598 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:23:52.237556 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd697ac_eb1a_4cda_a72a_6024967509f4.slice/crio-d8d76b39ac399bf4b4111026333ad2cb2c97e340b15baf8a86b1af82569a78a3 WatchSource:0}: Error finding container d8d76b39ac399bf4b4111026333ad2cb2c97e340b15baf8a86b1af82569a78a3: Status 404 returned error can't find the container with id d8d76b39ac399bf4b4111026333ad2cb2c97e340b15baf8a86b1af82569a78a3 Jul 06 19:23:52.278672 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.278621 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sx4t5"] Jul 06 19:23:52.282938 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:23:52.282896 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79911034_6ae1_42d7_9b8a_4f370e0b3099.slice/crio-5a6ac72c31b40eea199f058791cdc2cb5a89b44033cff543498a236368be6c9a WatchSource:0}: Error finding container 5a6ac72c31b40eea199f058791cdc2cb5a89b44033cff543498a236368be6c9a: Status 404 returned error can't find the container with id 5a6ac72c31b40eea199f058791cdc2cb5a89b44033cff543498a236368be6c9a Jul 06 19:23:52.335095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.335063 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-czqtk"] Jul 06 19:23:52.340792 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:23:52.340762 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75ebb1d_b187_4fb5_b4bf_5cb7cbe217f4.slice/crio-d15fb1617b79d373fcfe990545baffaee9051d7c8d2c3c0cc9801b8e1c268b77 WatchSource:0}: Error finding container d15fb1617b79d373fcfe990545baffaee9051d7c8d2c3c0cc9801b8e1c268b77: Status 404 returned error can't find the container with id d15fb1617b79d373fcfe990545baffaee9051d7c8d2c3c0cc9801b8e1c268b77 Jul 06 19:23:52.640108 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.640065 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" event={"ID":"65362076-a446-4576-ad83-6a994035a453","Type":"ContainerStarted","Data":"001220e91b159c2b03b7a83d3ad988a8ef759cf94e01b00bc77cdd34b97aeffe"} Jul 06 19:23:52.640616 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.640127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" event={"ID":"65362076-a446-4576-ad83-6a994035a453","Type":"ContainerStarted","Data":"83f1f4547b7a106740cb87a2343249609ced525792efcfc54f461322a4ab8dd3"} Jul 06 19:23:52.640616 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.640246 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:23:52.641266 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.641232 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sx4t5" event={"ID":"79911034-6ae1-42d7-9b8a-4f370e0b3099","Type":"ContainerStarted","Data":"5a6ac72c31b40eea199f058791cdc2cb5a89b44033cff543498a236368be6c9a"} Jul 06 19:23:52.642469 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.642439 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czqtk" event={"ID":"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4","Type":"ContainerStarted","Data":"d15fb1617b79d373fcfe990545baffaee9051d7c8d2c3c0cc9801b8e1c268b77"} Jul 06 19:23:52.644266 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.644236 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v85qf" event={"ID":"8bd697ac-eb1a-4cda-a72a-6024967509f4","Type":"ContainerStarted","Data":"d8d76b39ac399bf4b4111026333ad2cb2c97e340b15baf8a86b1af82569a78a3"} Jul 06 19:23:52.664553 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:52.664503 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" podStartSLOduration=1.664482244 podStartE2EDuration="1.664482244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:52.663921254 +0000 UTC m=+56.761149013" watchObservedRunningTime="2026-07-06 19:23:52.664482244 +0000 UTC m=+56.761710000" Jul 06 19:23:57.582012 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:57.581983 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q84g6" Jul 06 19:23:57.661516 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:57.661450 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v85qf" event={"ID":"8bd697ac-eb1a-4cda-a72a-6024967509f4","Type":"ContainerStarted","Data":"623a23a81150c6d9d9f79ec2014d1d479b339c93fa6b99fdb941aa78584ce843"} Jul 06 19:23:57.667808 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:57.667536 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sx4t5" event={"ID":"79911034-6ae1-42d7-9b8a-4f370e0b3099","Type":"ContainerStarted","Data":"5d8866de9caa985b5384aab62bed046e9a0110dceea43c38656e6789c4943ae4"} Jul 06 19:23:57.672406 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:57.672335 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czqtk" event={"ID":"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4","Type":"ContainerStarted","Data":"bdde208bb3bb6780ce90967052490c09a9f3f659f552e4d32ad26826523a2155"} Jul 06 19:23:57.702348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:57.702308 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v85qf" podStartSLOduration=1.418428451 podStartE2EDuration="6.702288259s" totalImagesPullingTime="5.283859808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:57.701905709 +0000 UTC m=+61.799133463" watchObservedRunningTime="2026-07-06 19:23:57.702288259 +0000 UTC m=+61.799516012" Jul 06 19:23:58.678463 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:58.678425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sx4t5" event={"ID":"79911034-6ae1-42d7-9b8a-4f370e0b3099","Type":"ContainerStarted","Data":"5bb55bbb3b86396f7ac286939ef2c5d613b9c7c909fea88a3b273575b3e0baf2"} Jul 06 19:23:58.678908 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:58.678509 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sx4t5" Jul 06 19:23:58.680650 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:58.680619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czqtk" event={"ID":"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4","Type":"ContainerStarted","Data":"b1864e958c6038f37820b923e8b7c0e477db12919ca5346f7d40f9fa091b136d"} Jul 06 19:23:58.707949 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:23:58.707898 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sx4t5" podStartSLOduration=2.4687780679999998 podStartE2EDuration="7.707882684s" totalImagesPullingTime="5.239104616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:58.706909662 +0000 UTC m=+62.804137428" watchObservedRunningTime="2026-07-06 19:23:58.707882684 +0000 UTC m=+62.805110444" Jul 06 19:24:00.688141 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.688105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czqtk" event={"ID":"c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4","Type":"ContainerStarted","Data":"124742b2bfb06252b24495bcec872e416acff426093d9e2bfedb5fd6d383a029"} Jul 06 19:24:00.715376 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.715322 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-czqtk" podStartSLOduration=1.5477137029999999 podStartE2EDuration="9.715305847s" totalImagesPullingTime="8.167592144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:00.71473816 +0000 UTC m=+64.811965913" watchObservedRunningTime="2026-07-06 19:24:00.715305847 +0000 UTC m=+64.812533601" Jul 06 19:24:00.863067 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.862995 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dfc85b48f-svkjb"] Jul 06 19:24:00.864825 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.864809 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:00.868376 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.868354 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:24:00.868592 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.868576 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:24:00.868592 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.868581 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:24:00.868828 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.868813 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:24:00.869115 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.869096 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:24:00.869340 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.869317 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8v4fg\"" Jul 06 19:24:00.869453 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.869323 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:24:00.869543 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.869531 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:24:00.899915 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.899886 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:24:00.902821 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.902800 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dfc85b48f-svkjb"] Jul 06 19:24:00.943260 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.943222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-oauth-serving-cert\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:00.943431 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.943264 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-service-ca\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:00.943431 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.943292 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xj9\" (UniqueName: \"kubernetes.io/projected/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-kube-api-access-99xj9\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:00.943431 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.943322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-oauth-config\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:00.943551 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.943426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-serving-cert\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:00.943551 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.943455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-config\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:00.943551 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:00.943477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-trusted-ca-bundle\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.044021 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.043980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-serving-cert\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.044021 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.044018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-config\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.044297 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.044034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-trusted-ca-bundle\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.044297 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.044067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-oauth-serving-cert\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.044297 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.044092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-service-ca\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.044297 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.044120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xj9\" (UniqueName: \"kubernetes.io/projected/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-kube-api-access-99xj9\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.044297 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.044155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-oauth-config\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.044920 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.044892 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-config\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.045060 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.044922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-service-ca\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.045060 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.044901 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-oauth-serving-cert\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.045178 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.045127 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-trusted-ca-bundle\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.046579 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.046550 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-serving-cert\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.046724 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.046706 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-oauth-config\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.052902 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.052882 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xj9\" (UniqueName: \"kubernetes.io/projected/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-kube-api-access-99xj9\") pod \"console-7dfc85b48f-svkjb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.173580 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.173496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:01.309671 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.309636 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dfc85b48f-svkjb"] Jul 06 19:24:01.312302 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:24:01.312277 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833a01e2_8fbe_4203_926f_f8e3e4f2bbdb.slice/crio-153c5505e03e1bf62d823d68b55ebe49cd2c93456592d6652ba8be12e4fdd8c0 WatchSource:0}: Error finding container 153c5505e03e1bf62d823d68b55ebe49cd2c93456592d6652ba8be12e4fdd8c0: Status 404 returned error can't find the container with id 153c5505e03e1bf62d823d68b55ebe49cd2c93456592d6652ba8be12e4fdd8c0 Jul 06 19:24:01.692484 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:01.692443 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dfc85b48f-svkjb" event={"ID":"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb","Type":"ContainerStarted","Data":"153c5505e03e1bf62d823d68b55ebe49cd2c93456592d6652ba8be12e4fdd8c0"} Jul 06 19:24:02.052392 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.052296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:24:02.054720 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.054696 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:24:02.065651 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.065628 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4-metrics-certs\") pod \"network-metrics-daemon-prgr9\" (UID: \"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4\") " pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:24:02.153251 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.153192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:24:02.153436 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.153316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:24:02.156892 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.156865 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:24:02.157395 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.157372 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:24:02.166262 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.166225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f20bf0b-a1f4-43f3-800d-a892bff4c7fe-original-pull-secret\") pod \"global-pull-secret-syncer-bsvw4\" (UID: \"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe\") " pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:24:02.169408 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.169385 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:24:02.176943 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.176921 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f968j\" (UniqueName: \"kubernetes.io/projected/72b43de8-9bee-4860-bd8e-edc25cde27fd-kube-api-access-f968j\") pod \"network-check-target-9sjpt\" (UID: \"72b43de8-9bee-4860-bd8e-edc25cde27fd\") " pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:24:02.306122 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.306037 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bsvw4" Jul 06 19:24:02.315574 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.315551 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tr4sh\"" Jul 06 19:24:02.320959 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.320938 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d9llp\"" Jul 06 19:24:02.323613 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.323590 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:24:02.329033 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.329018 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prgr9" Jul 06 19:24:02.496706 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.496627 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bsvw4"] Jul 06 19:24:02.497936 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.497912 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-prgr9"] Jul 06 19:24:02.498736 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:24:02.498708 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f09b3d0_f583_4ef3_a8eb_47ebb6090ef4.slice/crio-7b0bf45dc71ee4f81cd003d16e3b459370badb58d67d24adcbb2da0d140131bf WatchSource:0}: Error finding container 7b0bf45dc71ee4f81cd003d16e3b459370badb58d67d24adcbb2da0d140131bf: Status 404 returned error can't find the container with id 7b0bf45dc71ee4f81cd003d16e3b459370badb58d67d24adcbb2da0d140131bf Jul 06 19:24:02.499508 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:24:02.499471 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f20bf0b_a1f4_43f3_800d_a892bff4c7fe.slice/crio-29a764b0a5a25c8ae5623a30818e2dc0f3825366787078c1078609c1b346786b WatchSource:0}: Error finding container 29a764b0a5a25c8ae5623a30818e2dc0f3825366787078c1078609c1b346786b: Status 404 returned error can't find the container with id 29a764b0a5a25c8ae5623a30818e2dc0f3825366787078c1078609c1b346786b Jul 06 19:24:02.514096 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.514071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9sjpt"] Jul 06 19:24:02.517277 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:24:02.517200 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b43de8_9bee_4860_bd8e_edc25cde27fd.slice/crio-cb70da02e97b93c2d8fd34d77708147e4c20e1aed87e8770310e9dbb4cb4a8e7 WatchSource:0}: Error finding container cb70da02e97b93c2d8fd34d77708147e4c20e1aed87e8770310e9dbb4cb4a8e7: Status 404 returned error can't find the container with id cb70da02e97b93c2d8fd34d77708147e4c20e1aed87e8770310e9dbb4cb4a8e7 Jul 06 19:24:02.603806 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.603771 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj"] Jul 06 19:24:02.607382 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.607359 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.612007 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.611984 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:24:02.612654 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.612631 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:24:02.612654 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.612647 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 19:24:02.612810 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.612694 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7sjpz\"" Jul 06 19:24:02.612810 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.612700 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:24:02.614368 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.614349 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:24:02.623175 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.623152 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj"] Jul 06 19:24:02.657137 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.657104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.657137 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.657145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.657381 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.657166 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.657381 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.657237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdk94\" (UniqueName: \"kubernetes.io/projected/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-kube-api-access-xdk94\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.660616 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.660593 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-842r7"] Jul 06 19:24:02.663690 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.663672 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.666223 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.666156 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:24:02.666328 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.666246 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:24:02.666387 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.666247 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:24:02.666640 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.666614 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4jjw4\"" Jul 06 19:24:02.704299 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.704239 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bsvw4" event={"ID":"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe","Type":"ContainerStarted","Data":"29a764b0a5a25c8ae5623a30818e2dc0f3825366787078c1078609c1b346786b"} Jul 06 19:24:02.705636 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.705604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prgr9" event={"ID":"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4","Type":"ContainerStarted","Data":"7b0bf45dc71ee4f81cd003d16e3b459370badb58d67d24adcbb2da0d140131bf"} Jul 06 19:24:02.706898 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.706870 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9sjpt" event={"ID":"72b43de8-9bee-4860-bd8e-edc25cde27fd","Type":"ContainerStarted","Data":"cb70da02e97b93c2d8fd34d77708147e4c20e1aed87e8770310e9dbb4cb4a8e7"} Jul 06 19:24:02.758106 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758063 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2efca560-1b2f-45e1-93e7-75a60e94d465-sys\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.758288 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.758288 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2efca560-1b2f-45e1-93e7-75a60e94d465-root\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.758402 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.758451 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9wlf\" (UniqueName: \"kubernetes.io/projected/2efca560-1b2f-45e1-93e7-75a60e94d465-kube-api-access-t9wlf\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.758501 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.758501 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2efca560-1b2f-45e1-93e7-75a60e94d465-metrics-client-ca\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.758645 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-textfile\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.758645 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.758645 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdk94\" (UniqueName: \"kubernetes.io/projected/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-kube-api-access-xdk94\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.758645 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-accelerators-collector-config\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.758822 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-tls\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.758822 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.758747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-wtmp\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.759077 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.759060 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.760994 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.760967 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.761155 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.761134 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.769604 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.769576 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdk94\" (UniqueName: \"kubernetes.io/projected/06c1378b-4fb3-4b87-ac27-cd84f0a1793c-kube-api-access-xdk94\") pod \"openshift-state-metrics-54cf96dfcf-8ngjj\" (UID: \"06c1378b-4fb3-4b87-ac27-cd84f0a1793c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.859813 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2efca560-1b2f-45e1-93e7-75a60e94d465-sys\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.859813 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859769 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2efca560-1b2f-45e1-93e7-75a60e94d465-root\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.859813 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859805 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9wlf\" (UniqueName: \"kubernetes.io/projected/2efca560-1b2f-45e1-93e7-75a60e94d465-kube-api-access-t9wlf\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859812 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2efca560-1b2f-45e1-93e7-75a60e94d465-sys\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2efca560-1b2f-45e1-93e7-75a60e94d465-metrics-client-ca\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-textfile\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-accelerators-collector-config\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859961 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2efca560-1b2f-45e1-93e7-75a60e94d465-root\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.859988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-tls\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.860016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-wtmp\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860487 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.860145 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-wtmp\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860487 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.860303 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-textfile\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860728 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.860710 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-accelerators-collector-config\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.860803 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.860721 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2efca560-1b2f-45e1-93e7-75a60e94d465-metrics-client-ca\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.862600 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.862568 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.862691 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.862613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2efca560-1b2f-45e1-93e7-75a60e94d465-node-exporter-tls\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.882779 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.882744 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9wlf\" (UniqueName: \"kubernetes.io/projected/2efca560-1b2f-45e1-93e7-75a60e94d465-kube-api-access-t9wlf\") pod \"node-exporter-842r7\" (UID: \"2efca560-1b2f-45e1-93e7-75a60e94d465\") " pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.918579 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.918547 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" Jul 06 19:24:02.975383 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:02.975223 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-842r7" Jul 06 19:24:02.994590 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:24:02.994555 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efca560_1b2f_45e1_93e7_75a60e94d465.slice/crio-279ce7ecd618d263d2684ba340d883fa2989095012b479953d3ffd7fc13fa07a WatchSource:0}: Error finding container 279ce7ecd618d263d2684ba340d883fa2989095012b479953d3ffd7fc13fa07a: Status 404 returned error can't find the container with id 279ce7ecd618d263d2684ba340d883fa2989095012b479953d3ffd7fc13fa07a Jul 06 19:24:03.095342 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:03.095257 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj"] Jul 06 19:24:03.099264 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:24:03.099227 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c1378b_4fb3_4b87_ac27_cd84f0a1793c.slice/crio-f453d481eeedfca832a86fbe9fa9c01f86a51a66a006ff4eb6d9e60835c09c51 WatchSource:0}: Error finding container f453d481eeedfca832a86fbe9fa9c01f86a51a66a006ff4eb6d9e60835c09c51: Status 404 returned error can't find the container with id f453d481eeedfca832a86fbe9fa9c01f86a51a66a006ff4eb6d9e60835c09c51 Jul 06 19:24:03.713180 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:03.713138 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-842r7" event={"ID":"2efca560-1b2f-45e1-93e7-75a60e94d465","Type":"ContainerStarted","Data":"279ce7ecd618d263d2684ba340d883fa2989095012b479953d3ffd7fc13fa07a"} Jul 06 19:24:03.715515 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:03.715383 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" event={"ID":"06c1378b-4fb3-4b87-ac27-cd84f0a1793c","Type":"ContainerStarted","Data":"7ff0b9d8f647a7cdd5c0447dd9395d151b0b48be4f1088e211660fa36a42f6a1"} Jul 06 19:24:03.715515 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:03.715448 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" event={"ID":"06c1378b-4fb3-4b87-ac27-cd84f0a1793c","Type":"ContainerStarted","Data":"21eb82e38554d3371266c1fbabc37893caa4d4ac8cc171c1042da5fb36f709be"} Jul 06 19:24:03.715515 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:03.715478 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" event={"ID":"06c1378b-4fb3-4b87-ac27-cd84f0a1793c","Type":"ContainerStarted","Data":"f453d481eeedfca832a86fbe9fa9c01f86a51a66a006ff4eb6d9e60835c09c51"} Jul 06 19:24:04.764899 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.764725 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d689b697b-qd29t"] Jul 06 19:24:04.772635 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.772602 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.777009 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.776521 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 19:24:04.777009 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.776600 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 19:24:04.777009 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.776838 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 19:24:04.777009 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.776858 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5tldv\"" Jul 06 19:24:04.777499 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.777481 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 19:24:04.777631 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.777610 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 19:24:04.778336 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.778156 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eoideot7pockp\"" Jul 06 19:24:04.786979 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.786891 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d689b697b-qd29t"] Jul 06 19:24:04.877646 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.877610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-tls\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.877802 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.877659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm8vf\" (UniqueName: \"kubernetes.io/projected/55065c34-242e-474c-86f2-dbaf200768f1-kube-api-access-nm8vf\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.877802 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.877694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.877802 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.877734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.877802 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.877769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.877939 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.877809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55065c34-242e-474c-86f2-dbaf200768f1-metrics-client-ca\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.877939 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.877859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-grpc-tls\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.877939 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.877887 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.979953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.978807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-grpc-tls\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.979953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.978862 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.979953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.978910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-tls\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.979953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.978936 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm8vf\" (UniqueName: \"kubernetes.io/projected/55065c34-242e-474c-86f2-dbaf200768f1-kube-api-access-nm8vf\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.979953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.978971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.979953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.979003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.979953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.979034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.979953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.979073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55065c34-242e-474c-86f2-dbaf200768f1-metrics-client-ca\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.979953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.979893 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55065c34-242e-474c-86f2-dbaf200768f1-metrics-client-ca\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.982530 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.982504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.984115 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.984063 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.984601 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.984579 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-grpc-tls\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.984708 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.984653 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.985131 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.985092 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-tls\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.985703 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.985683 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/55065c34-242e-474c-86f2-dbaf200768f1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:04.991733 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:04.991689 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm8vf\" (UniqueName: \"kubernetes.io/projected/55065c34-242e-474c-86f2-dbaf200768f1-kube-api-access-nm8vf\") pod \"thanos-querier-7d689b697b-qd29t\" (UID: \"55065c34-242e-474c-86f2-dbaf200768f1\") " pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:05.086155 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:05.086099 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:07.164688 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.164649 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7c4684648-cmjsv"] Jul 06 19:24:07.167917 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.167893 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.172148 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.172124 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 19:24:07.172278 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.172123 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4p6bjanf457kr\"" Jul 06 19:24:07.172278 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.172162 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 19:24:07.172529 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.172506 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 19:24:07.172631 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.172533 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 19:24:07.172706 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.172652 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7qr46\"" Jul 06 19:24:07.189000 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.188977 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c4684648-cmjsv"] Jul 06 19:24:07.198340 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.198318 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-secret-metrics-server-client-certs\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.198461 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.198351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-metrics-server-audit-profiles\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.198461 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.198389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-client-ca-bundle\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.198541 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.198476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-secret-metrics-server-tls\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.198541 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.198511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.198634 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.198545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-audit-log\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.198634 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.198626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chdn\" (UniqueName: \"kubernetes.io/projected/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-kube-api-access-2chdn\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.299257 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.299201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-secret-metrics-server-tls\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.299423 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.299269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.299423 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.299300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-audit-log\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.299423 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.299329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chdn\" (UniqueName: \"kubernetes.io/projected/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-kube-api-access-2chdn\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.299423 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.299391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-secret-metrics-server-client-certs\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.299423 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.299416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-metrics-server-audit-profiles\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.299654 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.299445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-client-ca-bundle\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.299959 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.299908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-audit-log\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.300412 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.300360 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.300551 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.300525 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-metrics-server-audit-profiles\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.302101 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.302072 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-client-ca-bundle\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.302199 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.302129 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-secret-metrics-server-client-certs\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.302199 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.302189 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-secret-metrics-server-tls\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.350500 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.350458 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chdn\" (UniqueName: \"kubernetes.io/projected/fff59c3d-13b5-42cb-b3ea-6efcd180b04c-kube-api-access-2chdn\") pod \"metrics-server-7c4684648-cmjsv\" (UID: \"fff59c3d-13b5-42cb-b3ea-6efcd180b04c\") " pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:07.478261 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:07.478155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:08.011061 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.010891 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d689b697b-qd29t"] Jul 06 19:24:08.021836 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:24:08.021791 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55065c34_242e_474c_86f2_dbaf200768f1.slice/crio-8e0c20abce6f5df2f794c335867d3c60bcf5358856aeea441201fa7437198fb1 WatchSource:0}: Error finding container 8e0c20abce6f5df2f794c335867d3c60bcf5358856aeea441201fa7437198fb1: Status 404 returned error can't find the container with id 8e0c20abce6f5df2f794c335867d3c60bcf5358856aeea441201fa7437198fb1 Jul 06 19:24:08.036382 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.036284 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c4684648-cmjsv"] Jul 06 19:24:08.041178 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:24:08.041148 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff59c3d_13b5_42cb_b3ea_6efcd180b04c.slice/crio-5c1417b8ceaa28ba83eaf97c6e621547f3ae86bd05853f8fe553a36ea78ad09e WatchSource:0}: Error finding container 5c1417b8ceaa28ba83eaf97c6e621547f3ae86bd05853f8fe553a36ea78ad09e: Status 404 returned error can't find the container with id 5c1417b8ceaa28ba83eaf97c6e621547f3ae86bd05853f8fe553a36ea78ad09e Jul 06 19:24:08.686024 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.685995 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sx4t5" Jul 06 19:24:08.736915 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.736876 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bsvw4" event={"ID":"6f20bf0b-a1f4-43f3-800d-a892bff4c7fe","Type":"ContainerStarted","Data":"06fbf23cdd087ac3f791bc852e498f5836fa7a69af672fccc06ad0f60b5e9953"} Jul 06 19:24:08.738616 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.738589 2573 generic.go:356] "Generic (PLEG): container finished" podID="2efca560-1b2f-45e1-93e7-75a60e94d465" containerID="5265f36b6595962c8a694148a8be824a423f2685072433f5eb6a6ccad2b4a1b6" exitCode=0 Jul 06 19:24:08.738750 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.738666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-842r7" event={"ID":"2efca560-1b2f-45e1-93e7-75a60e94d465","Type":"ContainerDied","Data":"5265f36b6595962c8a694148a8be824a423f2685072433f5eb6a6ccad2b4a1b6"} Jul 06 19:24:08.740564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.740545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prgr9" event={"ID":"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4","Type":"ContainerStarted","Data":"462d669fd6f0c927c70543092de9ee795bc08ef8c5f168f4790254a6fd57f9e0"} Jul 06 19:24:08.740663 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.740595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prgr9" event={"ID":"7f09b3d0-f583-4ef3-a8eb-47ebb6090ef4","Type":"ContainerStarted","Data":"c5adc4e369b18a0bcc32d46704de5cb5b926c53bc0fd717fd07721ac4c8f5646"} Jul 06 19:24:08.742918 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.742892 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" event={"ID":"06c1378b-4fb3-4b87-ac27-cd84f0a1793c","Type":"ContainerStarted","Data":"9146209357c063ca458179865783f403653eafdb4c59de24ef911e84279488e5"} Jul 06 19:24:08.744032 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.744003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" event={"ID":"fff59c3d-13b5-42cb-b3ea-6efcd180b04c","Type":"ContainerStarted","Data":"5c1417b8ceaa28ba83eaf97c6e621547f3ae86bd05853f8fe553a36ea78ad09e"} Jul 06 19:24:08.745386 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.745365 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dfc85b48f-svkjb" event={"ID":"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb","Type":"ContainerStarted","Data":"d2b1809dfdea1ee59a2e69d1e342cb1bf9128b53bbbbe68ca90ac95183911af4"} Jul 06 19:24:08.747031 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.747009 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9sjpt" event={"ID":"72b43de8-9bee-4860-bd8e-edc25cde27fd","Type":"ContainerStarted","Data":"74557cdc0007e68e68a57de85c23e5ab679569458c6068a34178f21150c5516a"} Jul 06 19:24:08.747119 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.747088 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:24:08.748107 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.748089 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" event={"ID":"55065c34-242e-474c-86f2-dbaf200768f1","Type":"ContainerStarted","Data":"8e0c20abce6f5df2f794c335867d3c60bcf5358856aeea441201fa7437198fb1"} Jul 06 19:24:08.763775 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.763724 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bsvw4" podStartSLOduration=66.430533753 podStartE2EDuration="1m11.763708219s" totalImagesPullingTime="5.333174466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:08.762337464 +0000 UTC m=+72.859565218" watchObservedRunningTime="2026-07-06 19:24:08.763708219 +0000 UTC m=+72.860935952" Jul 06 19:24:08.780962 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.780920 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9sjpt" podStartSLOduration=66.471537035 podStartE2EDuration="1m11.780902088s" totalImagesPullingTime="5.309365053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:08.779331009 +0000 UTC m=+72.876558763" watchObservedRunningTime="2026-07-06 19:24:08.780902088 +0000 UTC m=+72.878129869" Jul 06 19:24:08.801975 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.801922 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dfc85b48f-svkjb" podStartSLOduration=2.332261292 podStartE2EDuration="8.801901608s" totalImagesPullingTime="6.469640316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:08.801042122 +0000 UTC m=+72.898269875" watchObservedRunningTime="2026-07-06 19:24:08.801901608 +0000 UTC m=+72.899129362" Jul 06 19:24:08.849568 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.849504 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-prgr9" podStartSLOduration=68.110557486 podStartE2EDuration="1m11.849484002s" totalImagesPullingTime="3.738926516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:08.848767782 +0000 UTC m=+72.945995538" watchObservedRunningTime="2026-07-06 19:24:08.849484002 +0000 UTC m=+72.946711757" Jul 06 19:24:08.871559 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:08.871506 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8ngjj" podStartSLOduration=2.315796849 podStartE2EDuration="6.871490569s" totalImagesPullingTime="4.55569372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:08.870304656 +0000 UTC m=+72.967532415" watchObservedRunningTime="2026-07-06 19:24:08.871490569 +0000 UTC m=+72.968718322" Jul 06 19:24:09.757115 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:09.757078 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-842r7" event={"ID":"2efca560-1b2f-45e1-93e7-75a60e94d465","Type":"ContainerStarted","Data":"4a925ebe8d8507572a26d2ccb9113acd46d26409434b66275c71f398534d712d"} Jul 06 19:24:09.757608 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:09.757139 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-842r7" event={"ID":"2efca560-1b2f-45e1-93e7-75a60e94d465","Type":"ContainerStarted","Data":"75a802b04022ca41ed3438b90817eaf3aa5964dadb0e02cdded0ec6e457d31a6"} Jul 06 19:24:09.782388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:09.782340 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-842r7" podStartSLOduration=2.8662415169999997 podStartE2EDuration="7.782325715s" totalImagesPullingTime="4.824903784s" totalInitContainerRuntime="91.180414ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:09.782245734 +0000 UTC m=+73.879473490" watchObservedRunningTime="2026-07-06 19:24:09.782325715 +0000 UTC m=+73.879553507" Jul 06 19:24:10.762604 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:10.762505 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" event={"ID":"fff59c3d-13b5-42cb-b3ea-6efcd180b04c","Type":"ContainerStarted","Data":"540a85cc6420d4601525cd4b8cd6f1cec7047ecc908c117a1929f255f6a7b4a3"} Jul 06 19:24:10.764667 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:10.764644 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" event={"ID":"55065c34-242e-474c-86f2-dbaf200768f1","Type":"ContainerStarted","Data":"c1bb85789819754e6f42adee66c7745060938a627fec3119d551d8ea2de05a05"} Jul 06 19:24:10.764771 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:10.764698 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" event={"ID":"55065c34-242e-474c-86f2-dbaf200768f1","Type":"ContainerStarted","Data":"9357b9da101880b3557d61e130e9b40a77f452d9cf58fa4e4fb2885eca8c4c07"} Jul 06 19:24:10.764771 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:10.764728 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" event={"ID":"55065c34-242e-474c-86f2-dbaf200768f1","Type":"ContainerStarted","Data":"a831abef49f7c52acc90ee537409d2e18b3969173819d657f1aac76a750d0c74"} Jul 06 19:24:10.786118 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:10.786070 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" podStartSLOduration=1.435744254 podStartE2EDuration="3.786054663s" totalImagesPullingTime="2.350310409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:10.785548936 +0000 UTC m=+74.882776690" watchObservedRunningTime="2026-07-06 19:24:10.786054663 +0000 UTC m=+74.883282417" Jul 06 19:24:11.174468 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:11.174429 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:11.174647 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:11.174494 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:11.179657 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:11.179630 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:11.770848 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:11.770813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" event={"ID":"55065c34-242e-474c-86f2-dbaf200768f1","Type":"ContainerStarted","Data":"e287108d6657c657dbf115e3a92a16ee815b8ef02f8ae9c1b3bc06852b00409d"} Jul 06 19:24:11.771321 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:11.770887 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" event={"ID":"55065c34-242e-474c-86f2-dbaf200768f1","Type":"ContainerStarted","Data":"17562ea02a611f07b5e25c2cd8f03ce0b59059452d7166a85c3a584597409649"} Jul 06 19:24:11.771321 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:11.770926 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" event={"ID":"55065c34-242e-474c-86f2-dbaf200768f1","Type":"ContainerStarted","Data":"c5112d9145f73b8ad960d8678e4522e1e3022d66f20a33ada903afc05ae10a6e"} Jul 06 19:24:11.774843 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:11.774824 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:11.809166 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:11.809109 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" podStartSLOduration=4.7654554959999995 podStartE2EDuration="7.809091955s" totalImagesPullingTime="3.043636459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:11.805948297 +0000 UTC m=+75.903176064" watchObservedRunningTime="2026-07-06 19:24:11.809091955 +0000 UTC m=+75.906319708" Jul 06 19:24:12.775129 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:12.775101 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:13.652222 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:13.652178 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f59cc6f6d-bnxlq" Jul 06 19:24:18.428713 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:18.428593 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dfc85b48f-svkjb"] Jul 06 19:24:18.784312 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:18.784229 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d689b697b-qd29t" Jul 06 19:24:27.479051 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:27.479018 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:27.479462 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:27.479078 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:39.761032 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:39.760998 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9sjpt" Jul 06 19:24:43.448283 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:43.448246 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7dfc85b48f-svkjb" podUID="833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" containerName="console" containerID="cri-o://d2b1809dfdea1ee59a2e69d1e342cb1bf9128b53bbbbe68ca90ac95183911af4" gracePeriod=15 Jul 06 19:24:43.887009 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:43.886978 2573 generic.go:356] "Generic (PLEG): container finished" podID="833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" containerID="d2b1809dfdea1ee59a2e69d1e342cb1bf9128b53bbbbe68ca90ac95183911af4" exitCode=0 Jul 06 19:24:43.887185 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:43.887059 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dfc85b48f-svkjb" event={"ID":"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb","Type":"ContainerDied","Data":"d2b1809dfdea1ee59a2e69d1e342cb1bf9128b53bbbbe68ca90ac95183911af4"} Jul 06 19:24:44.202796 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.202770 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:44.296284 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296249 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99xj9\" (UniqueName: \"kubernetes.io/projected/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-kube-api-access-99xj9\") pod \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " Jul 06 19:24:44.296485 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296301 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-oauth-serving-cert\") pod \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " Jul 06 19:24:44.296485 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296324 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-serving-cert\") pod \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " Jul 06 19:24:44.296485 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296340 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-oauth-config\") pod \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " Jul 06 19:24:44.296485 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296371 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-trusted-ca-bundle\") pod \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " Jul 06 19:24:44.296485 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296399 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-service-ca\") pod \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " Jul 06 19:24:44.296485 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296475 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-config\") pod \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\" (UID: \"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb\") " Jul 06 19:24:44.296872 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296828 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" (UID: "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:44.296935 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296876 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" (UID: "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:44.296988 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296937 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-config" (OuterVolumeSpecName: "console-config") pod "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" (UID: "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:44.297045 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.296990 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-service-ca" (OuterVolumeSpecName: "service-ca") pod "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" (UID: "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:44.298716 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.298692 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" (UID: "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:24:44.298850 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.298828 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" (UID: "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:24:44.298915 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.298880 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-kube-api-access-99xj9" (OuterVolumeSpecName: "kube-api-access-99xj9") pod "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" (UID: "833a01e2-8fbe-4203-926f-f8e3e4f2bbdb"). InnerVolumeSpecName "kube-api-access-99xj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:44.396944 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.396919 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:24:44.396944 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.396939 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:24:44.396944 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.396949 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xj9\" (UniqueName: \"kubernetes.io/projected/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-kube-api-access-99xj9\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:24:44.397154 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.396959 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-oauth-serving-cert\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:24:44.397154 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.396967 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-serving-cert\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:24:44.397154 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.396975 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-console-oauth-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:24:44.397154 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.396984 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb-trusted-ca-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:24:44.892181 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.892150 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dfc85b48f-svkjb" Jul 06 19:24:44.892630 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.892151 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dfc85b48f-svkjb" event={"ID":"833a01e2-8fbe-4203-926f-f8e3e4f2bbdb","Type":"ContainerDied","Data":"153c5505e03e1bf62d823d68b55ebe49cd2c93456592d6652ba8be12e4fdd8c0"} Jul 06 19:24:44.892630 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.892278 2573 scope.go:117] "RemoveContainer" containerID="d2b1809dfdea1ee59a2e69d1e342cb1bf9128b53bbbbe68ca90ac95183911af4" Jul 06 19:24:44.926494 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.926462 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dfc85b48f-svkjb"] Jul 06 19:24:44.930438 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:44.930419 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dfc85b48f-svkjb"] Jul 06 19:24:46.392802 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:46.392769 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" path="/var/lib/kubelet/pods/833a01e2-8fbe-4203-926f-f8e3e4f2bbdb/volumes" Jul 06 19:24:47.484461 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:47.484430 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:24:47.488100 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:24:47.488076 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7c4684648-cmjsv" Jul 06 19:25:27.206891 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.206855 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-f86dccc9f-zwmvg"] Jul 06 19:25:27.207411 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.207132 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" containerName="console" Jul 06 19:25:27.207411 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.207142 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" containerName="console" Jul 06 19:25:27.207411 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.207194 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="833a01e2-8fbe-4203-926f-f8e3e4f2bbdb" containerName="console" Jul 06 19:25:27.209285 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.209269 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.214656 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.214631 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 19:25:27.214656 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.214695 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ws8xc\"" Jul 06 19:25:27.214966 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.214741 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 19:25:27.214966 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.214924 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 19:25:27.214966 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.214938 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 19:25:27.215611 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.214976 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 19:25:27.222191 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.222172 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 19:25:27.226126 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.226101 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f86dccc9f-zwmvg"] Jul 06 19:25:27.295893 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.295855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-telemeter-client-tls\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.296095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.295905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf97e2c-9b20-42b0-ac71-79b006de51dc-serving-certs-ca-bundle\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.296095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.295935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-federate-client-tls\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.296095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.295981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.296095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.296028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf97e2c-9b20-42b0-ac71-79b006de51dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.296095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.296069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtqf\" (UniqueName: \"kubernetes.io/projected/5cf97e2c-9b20-42b0-ac71-79b006de51dc-kube-api-access-nrtqf\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.296095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.296089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-secret-telemeter-client\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.296352 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.296106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf97e2c-9b20-42b0-ac71-79b006de51dc-metrics-client-ca\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.397302 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.397260 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf97e2c-9b20-42b0-ac71-79b006de51dc-serving-certs-ca-bundle\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.397302 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.397306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-federate-client-tls\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.397509 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.397325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.397509 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.397388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf97e2c-9b20-42b0-ac71-79b006de51dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.397509 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.397428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtqf\" (UniqueName: \"kubernetes.io/projected/5cf97e2c-9b20-42b0-ac71-79b006de51dc-kube-api-access-nrtqf\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.397509 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.397451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-secret-telemeter-client\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.397509 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.397471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf97e2c-9b20-42b0-ac71-79b006de51dc-metrics-client-ca\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.397509 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.397504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-telemeter-client-tls\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.398276 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.398240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf97e2c-9b20-42b0-ac71-79b006de51dc-serving-certs-ca-bundle\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.398394 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.398296 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf97e2c-9b20-42b0-ac71-79b006de51dc-metrics-client-ca\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.398615 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.398593 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf97e2c-9b20-42b0-ac71-79b006de51dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.399905 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.399885 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.400114 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.400092 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-telemeter-client-tls\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.400281 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.400264 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-secret-telemeter-client\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.400327 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.400277 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5cf97e2c-9b20-42b0-ac71-79b006de51dc-federate-client-tls\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.405842 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.405819 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtqf\" (UniqueName: \"kubernetes.io/projected/5cf97e2c-9b20-42b0-ac71-79b006de51dc-kube-api-access-nrtqf\") pod \"telemeter-client-f86dccc9f-zwmvg\" (UID: \"5cf97e2c-9b20-42b0-ac71-79b006de51dc\") " pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.518376 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.518289 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" Jul 06 19:25:27.646231 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:27.646188 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f86dccc9f-zwmvg"] Jul 06 19:25:27.647603 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:25:27.647582 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf97e2c_9b20_42b0_ac71_79b006de51dc.slice/crio-94f8f528ec43164bb488a7f7a589fc7b7a0ee97dd8b74bfa140b099c1ad08067 WatchSource:0}: Error finding container 94f8f528ec43164bb488a7f7a589fc7b7a0ee97dd8b74bfa140b099c1ad08067: Status 404 returned error can't find the container with id 94f8f528ec43164bb488a7f7a589fc7b7a0ee97dd8b74bfa140b099c1ad08067 Jul 06 19:25:28.036618 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:28.036583 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" event={"ID":"5cf97e2c-9b20-42b0-ac71-79b006de51dc","Type":"ContainerStarted","Data":"94f8f528ec43164bb488a7f7a589fc7b7a0ee97dd8b74bfa140b099c1ad08067"} Jul 06 19:25:30.044529 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:30.044495 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" event={"ID":"5cf97e2c-9b20-42b0-ac71-79b006de51dc","Type":"ContainerStarted","Data":"012548599ff5e468a4db5fc37f43e7e062d65f01f2e9ec6c4a9349cad6eb241d"} Jul 06 19:25:31.049989 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.049951 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" event={"ID":"5cf97e2c-9b20-42b0-ac71-79b006de51dc","Type":"ContainerStarted","Data":"280fad6c9e62918a7f8cede5328f37740ad9f91bcf71e4ef92884042cf8f6648"} Jul 06 19:25:31.050408 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.050008 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" event={"ID":"5cf97e2c-9b20-42b0-ac71-79b006de51dc","Type":"ContainerStarted","Data":"16af488faebe080ac6654f640f2d6eb16ffb9697f2b9275096a9a34d24868043"} Jul 06 19:25:31.074412 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.074295 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-f86dccc9f-zwmvg" podStartSLOduration=0.963648773 podStartE2EDuration="4.074275644s" totalImagesPullingTime="3.110626871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:31.073227234 +0000 UTC m=+155.170454981" watchObservedRunningTime="2026-07-06 19:25:31.074275644 +0000 UTC m=+155.171503396" Jul 06 19:25:31.915129 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.915097 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58c9bf4d7f-dw5qr"] Jul 06 19:25:31.917341 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.917326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:31.919539 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.919517 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:25:31.920285 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.920268 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:25:31.920460 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.920312 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:25:31.920460 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.920321 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:25:31.920460 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.920365 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8v4fg\"" Jul 06 19:25:31.920460 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.920401 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:25:31.920460 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.920445 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:25:31.920460 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.920268 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:25:31.925687 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.925669 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:25:31.929562 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:31.929541 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c9bf4d7f-dw5qr"] Jul 06 19:25:32.036892 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.036857 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-oauth-config\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.037067 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.036904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-trusted-ca-bundle\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.037067 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.036926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-serving-cert\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.037067 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.036966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-oauth-serving-cert\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.037067 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.036993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-service-ca\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.037067 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.037049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-config\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.037067 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.037063 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fw4c\" (UniqueName: \"kubernetes.io/projected/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-kube-api-access-4fw4c\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.138336 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.138293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-trusted-ca-bundle\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.138336 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.138335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-serving-cert\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.138830 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.138353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-oauth-serving-cert\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.138830 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.138376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-service-ca\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.138830 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.138413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-config\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.138830 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.138427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fw4c\" (UniqueName: \"kubernetes.io/projected/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-kube-api-access-4fw4c\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.138830 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.138515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-oauth-config\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.139124 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.139095 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-oauth-serving-cert\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.139549 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.139527 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-config\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.139623 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.139550 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-trusted-ca-bundle\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.139749 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.139734 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-service-ca\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.141083 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.141051 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-oauth-config\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.141231 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.141199 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-serving-cert\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.148607 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.148585 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fw4c\" (UniqueName: \"kubernetes.io/projected/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-kube-api-access-4fw4c\") pod \"console-58c9bf4d7f-dw5qr\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.227574 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.227496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:32.352925 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:32.352901 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c9bf4d7f-dw5qr"] Jul 06 19:25:32.355012 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:25:32.354977 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce19a880_695e_45cd_a4e4_6a99bf0eb6cc.slice/crio-dc569268ddc14c5dc7266f310c3de68ac67609c7564961a6af1b8a3b7b6dc446 WatchSource:0}: Error finding container dc569268ddc14c5dc7266f310c3de68ac67609c7564961a6af1b8a3b7b6dc446: Status 404 returned error can't find the container with id dc569268ddc14c5dc7266f310c3de68ac67609c7564961a6af1b8a3b7b6dc446 Jul 06 19:25:33.062432 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:33.062398 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c9bf4d7f-dw5qr" event={"ID":"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc","Type":"ContainerStarted","Data":"1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2"} Jul 06 19:25:33.062741 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:33.062479 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c9bf4d7f-dw5qr" event={"ID":"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc","Type":"ContainerStarted","Data":"dc569268ddc14c5dc7266f310c3de68ac67609c7564961a6af1b8a3b7b6dc446"} Jul 06 19:25:33.082828 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:33.082782 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58c9bf4d7f-dw5qr" podStartSLOduration=2.082767299 podStartE2EDuration="2.082767299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:33.08105586 +0000 UTC m=+157.178283617" watchObservedRunningTime="2026-07-06 19:25:33.082767299 +0000 UTC m=+157.179995051" Jul 06 19:25:42.227829 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:42.227795 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:42.228319 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:42.227856 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:42.232505 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:42.232482 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:25:43.100406 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:25:43.100382 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:26:43.096028 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.095995 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66846d6df4-fh6f4"] Jul 06 19:26:43.099131 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.099111 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.136966 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.136934 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66846d6df4-fh6f4"] Jul 06 19:26:43.191434 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.191402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-config\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.191612 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.191450 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-serving-cert\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.191612 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.191483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6475\" (UniqueName: \"kubernetes.io/projected/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-kube-api-access-s6475\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.191612 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.191513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-oauth-serving-cert\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.191612 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.191529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-trusted-ca-bundle\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.191612 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.191580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-service-ca\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.191612 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.191596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-oauth-config\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.292820 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.292784 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-service-ca\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.292820 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.292824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-oauth-config\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.293057 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.292847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-config\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.293057 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.292888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-serving-cert\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.293057 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.292919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6475\" (UniqueName: \"kubernetes.io/projected/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-kube-api-access-s6475\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.293057 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.292941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-oauth-serving-cert\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.293057 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.292969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-trusted-ca-bundle\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.293732 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.293703 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-service-ca\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.293868 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.293729 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-oauth-serving-cert\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.293868 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.293737 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-config\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.294278 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.294256 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-trusted-ca-bundle\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.295219 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.295192 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-oauth-config\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.295421 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.295406 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-serving-cert\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.301591 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.301567 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6475\" (UniqueName: \"kubernetes.io/projected/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-kube-api-access-s6475\") pod \"console-66846d6df4-fh6f4\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.408788 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.408685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:43.532826 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:43.532803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66846d6df4-fh6f4"] Jul 06 19:26:43.534874 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:26:43.534834 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bde0ff7_a45f_4cc2_8372_01d0e271a46c.slice/crio-c1648ce72098ec389f1ccd654326706cff6f526b8134f8f60b154aca3405b350 WatchSource:0}: Error finding container c1648ce72098ec389f1ccd654326706cff6f526b8134f8f60b154aca3405b350: Status 404 returned error can't find the container with id c1648ce72098ec389f1ccd654326706cff6f526b8134f8f60b154aca3405b350 Jul 06 19:26:44.306655 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:44.306623 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66846d6df4-fh6f4" event={"ID":"9bde0ff7-a45f-4cc2-8372-01d0e271a46c","Type":"ContainerStarted","Data":"c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267"} Jul 06 19:26:44.307019 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:44.306701 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66846d6df4-fh6f4" event={"ID":"9bde0ff7-a45f-4cc2-8372-01d0e271a46c","Type":"ContainerStarted","Data":"c1648ce72098ec389f1ccd654326706cff6f526b8134f8f60b154aca3405b350"} Jul 06 19:26:44.327813 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:44.327759 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66846d6df4-fh6f4" podStartSLOduration=1.327744292 podStartE2EDuration="1.327744292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:44.327417989 +0000 UTC m=+228.424645744" watchObservedRunningTime="2026-07-06 19:26:44.327744292 +0000 UTC m=+228.424972045" Jul 06 19:26:53.408951 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:53.408902 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:53.409347 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:53.408978 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:53.414072 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:53.414050 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:54.345418 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:54.345391 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:26:54.420884 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:26:54.420846 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c9bf4d7f-dw5qr"] Jul 06 19:27:13.928627 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:13.928593 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf"] Jul 06 19:27:13.931916 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:13.931900 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:13.934790 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:13.934771 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p5qdc\"" Jul 06 19:27:13.935093 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:13.935078 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:27:13.935607 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:13.935591 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:27:13.945582 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:13.945559 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf"] Jul 06 19:27:14.034745 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.034708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxv28\" (UniqueName: \"kubernetes.io/projected/07dce484-0b1d-40b0-9393-2aa0493db3c7-kube-api-access-pxv28\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.034924 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.034754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.034924 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.034802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.140691 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.140649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxv28\" (UniqueName: \"kubernetes.io/projected/07dce484-0b1d-40b0-9393-2aa0493db3c7-kube-api-access-pxv28\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.140866 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.140843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.140919 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.140906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.141227 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.141185 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.141350 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.141317 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.150167 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.150144 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxv28\" (UniqueName: \"kubernetes.io/projected/07dce484-0b1d-40b0-9393-2aa0493db3c7-kube-api-access-pxv28\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.240566 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.240471 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:14.370415 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.370389 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf"] Jul 06 19:27:14.371804 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:27:14.371777 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07dce484_0b1d_40b0_9393_2aa0493db3c7.slice/crio-829a09a267f53813da56689f8e12c28f9f88d4258ee267ab2f163a95227920b6 WatchSource:0}: Error finding container 829a09a267f53813da56689f8e12c28f9f88d4258ee267ab2f163a95227920b6: Status 404 returned error can't find the container with id 829a09a267f53813da56689f8e12c28f9f88d4258ee267ab2f163a95227920b6 Jul 06 19:27:14.407015 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:14.406981 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" event={"ID":"07dce484-0b1d-40b0-9393-2aa0493db3c7","Type":"ContainerStarted","Data":"829a09a267f53813da56689f8e12c28f9f88d4258ee267ab2f163a95227920b6"} Jul 06 19:27:19.445746 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.445633 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-58c9bf4d7f-dw5qr" podUID="ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" containerName="console" containerID="cri-o://1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2" gracePeriod=15 Jul 06 19:27:19.699521 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.699463 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:27:19.889834 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.889808 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-config\") pod \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " Jul 06 19:27:19.890003 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.889870 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-oauth-serving-cert\") pod \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " Jul 06 19:27:19.890003 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.889893 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-oauth-config\") pod \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " Jul 06 19:27:19.890003 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.889909 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-serving-cert\") pod \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " Jul 06 19:27:19.890003 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.889931 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fw4c\" (UniqueName: \"kubernetes.io/projected/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-kube-api-access-4fw4c\") pod \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " Jul 06 19:27:19.890003 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.889957 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-trusted-ca-bundle\") pod \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " Jul 06 19:27:19.890003 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.889980 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-service-ca\") pod \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\" (UID: \"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc\") " Jul 06 19:27:19.890296 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.890199 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-config" (OuterVolumeSpecName: "console-config") pod "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" (UID: "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:27:19.890296 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.890252 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" (UID: "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:27:19.890499 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.890467 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-service-ca" (OuterVolumeSpecName: "service-ca") pod "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" (UID: "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:27:19.890499 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.890482 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" (UID: "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:27:19.892124 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.892096 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" (UID: "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:27:19.892264 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.892119 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" (UID: "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:27:19.892264 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.892160 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-kube-api-access-4fw4c" (OuterVolumeSpecName: "kube-api-access-4fw4c") pod "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" (UID: "ce19a880-695e-45cd-a4e4-6a99bf0eb6cc"). InnerVolumeSpecName "kube-api-access-4fw4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:27:19.990970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.990889 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-oauth-serving-cert\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:19.990970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.990916 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-oauth-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:19.990970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.990925 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-serving-cert\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:19.990970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.990933 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fw4c\" (UniqueName: \"kubernetes.io/projected/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-kube-api-access-4fw4c\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:19.990970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.990941 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-trusted-ca-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:19.990970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.990950 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:19.990970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:19.990958 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc-console-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:20.430909 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:20.430877 2573 generic.go:356] "Generic (PLEG): container finished" podID="ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" containerID="1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2" exitCode=0 Jul 06 19:27:20.431095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:20.430940 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c9bf4d7f-dw5qr" Jul 06 19:27:20.431095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:20.430940 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c9bf4d7f-dw5qr" event={"ID":"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc","Type":"ContainerDied","Data":"1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2"} Jul 06 19:27:20.431095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:20.431005 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c9bf4d7f-dw5qr" event={"ID":"ce19a880-695e-45cd-a4e4-6a99bf0eb6cc","Type":"ContainerDied","Data":"dc569268ddc14c5dc7266f310c3de68ac67609c7564961a6af1b8a3b7b6dc446"} Jul 06 19:27:20.431095 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:20.431036 2573 scope.go:117] "RemoveContainer" containerID="1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2" Jul 06 19:27:20.456569 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:20.456537 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c9bf4d7f-dw5qr"] Jul 06 19:27:20.462253 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:20.462227 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58c9bf4d7f-dw5qr"] Jul 06 19:27:21.937466 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:21.937254 2573 scope.go:117] "RemoveContainer" containerID="1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2" Jul 06 19:27:21.937687 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:27:21.937563 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2\": container with ID starting with 1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2 not found: ID does not exist" containerID="1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2" Jul 06 19:27:21.937687 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:21.937602 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2"} err="failed to get container status \"1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2\": rpc error: code = NotFound desc = could not find container \"1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2\": container with ID starting with 1b33616fc5e4706d0315cb210849c6c9d59c98d3ebeb59d7093e771867d5d7c2 not found: ID does not exist" Jul 06 19:27:22.393280 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:22.393249 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" path="/var/lib/kubelet/pods/ce19a880-695e-45cd-a4e4-6a99bf0eb6cc/volumes" Jul 06 19:27:22.440296 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:22.440269 2573 generic.go:356] "Generic (PLEG): container finished" podID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerID="11cc0644e5a1d4352297de9c3da7c57e44c3050869c15c8171fad9d335413caf" exitCode=0 Jul 06 19:27:22.440469 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:22.440338 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" event={"ID":"07dce484-0b1d-40b0-9393-2aa0493db3c7","Type":"ContainerDied","Data":"11cc0644e5a1d4352297de9c3da7c57e44c3050869c15c8171fad9d335413caf"} Jul 06 19:27:25.452534 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:25.452498 2573 generic.go:356] "Generic (PLEG): container finished" podID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerID="e924a1ae63582072f9eab8c4943fce61034ed289960e4a6d3ea2b24a69f35966" exitCode=0 Jul 06 19:27:25.452932 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:25.452544 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" event={"ID":"07dce484-0b1d-40b0-9393-2aa0493db3c7","Type":"ContainerDied","Data":"e924a1ae63582072f9eab8c4943fce61034ed289960e4a6d3ea2b24a69f35966"} Jul 06 19:27:33.491519 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:33.491478 2573 generic.go:356] "Generic (PLEG): container finished" podID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerID="4aa19d18c5390912095b05cca5876c003ffda5c7519d848870bc435b7eb4c61c" exitCode=0 Jul 06 19:27:33.491889 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:33.491539 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" event={"ID":"07dce484-0b1d-40b0-9393-2aa0493db3c7","Type":"ContainerDied","Data":"4aa19d18c5390912095b05cca5876c003ffda5c7519d848870bc435b7eb4c61c"} Jul 06 19:27:34.614683 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.614660 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:34.715030 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.714993 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxv28\" (UniqueName: \"kubernetes.io/projected/07dce484-0b1d-40b0-9393-2aa0493db3c7-kube-api-access-pxv28\") pod \"07dce484-0b1d-40b0-9393-2aa0493db3c7\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " Jul 06 19:27:34.715199 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.715063 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-bundle\") pod \"07dce484-0b1d-40b0-9393-2aa0493db3c7\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " Jul 06 19:27:34.715280 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.715174 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-util\") pod \"07dce484-0b1d-40b0-9393-2aa0493db3c7\" (UID: \"07dce484-0b1d-40b0-9393-2aa0493db3c7\") " Jul 06 19:27:34.715789 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.715767 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-bundle" (OuterVolumeSpecName: "bundle") pod "07dce484-0b1d-40b0-9393-2aa0493db3c7" (UID: "07dce484-0b1d-40b0-9393-2aa0493db3c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:27:34.717358 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.717326 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07dce484-0b1d-40b0-9393-2aa0493db3c7-kube-api-access-pxv28" (OuterVolumeSpecName: "kube-api-access-pxv28") pod "07dce484-0b1d-40b0-9393-2aa0493db3c7" (UID: "07dce484-0b1d-40b0-9393-2aa0493db3c7"). InnerVolumeSpecName "kube-api-access-pxv28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:27:34.719811 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.719788 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-util" (OuterVolumeSpecName: "util") pod "07dce484-0b1d-40b0-9393-2aa0493db3c7" (UID: "07dce484-0b1d-40b0-9393-2aa0493db3c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:27:34.816744 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.816666 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:34.816744 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.816693 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07dce484-0b1d-40b0-9393-2aa0493db3c7-util\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:34.816744 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:34.816701 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxv28\" (UniqueName: \"kubernetes.io/projected/07dce484-0b1d-40b0-9393-2aa0493db3c7-kube-api-access-pxv28\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:35.500097 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:35.500065 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" Jul 06 19:27:35.500271 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:35.500066 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z5jnf" event={"ID":"07dce484-0b1d-40b0-9393-2aa0493db3c7","Type":"ContainerDied","Data":"829a09a267f53813da56689f8e12c28f9f88d4258ee267ab2f163a95227920b6"} Jul 06 19:27:35.500271 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:35.500178 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="829a09a267f53813da56689f8e12c28f9f88d4258ee267ab2f163a95227920b6" Jul 06 19:27:42.324334 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324302 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh"] Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324569 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerName="util" Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324581 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerName="util" Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324591 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" containerName="console" Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324599 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" containerName="console" Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324605 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerName="pull" Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324610 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerName="pull" Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324623 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerName="extract" Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324627 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerName="extract" Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324678 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce19a880-695e-45cd-a4e4-6a99bf0eb6cc" containerName="console" Jul 06 19:27:42.324763 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.324686 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07dce484-0b1d-40b0-9393-2aa0493db3c7" containerName="extract" Jul 06 19:27:42.328878 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.328859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" Jul 06 19:27:42.330931 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.330906 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:27:42.331057 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.330961 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mkkdz\"" Jul 06 19:27:42.331057 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.330990 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 19:27:42.340017 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.339995 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh"] Jul 06 19:27:42.374858 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.374830 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42e4403b-fc10-4515-9021-521d035885c1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-prcfh\" (UID: \"42e4403b-fc10-4515-9021-521d035885c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" Jul 06 19:27:42.375018 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.374883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bjvg\" (UniqueName: \"kubernetes.io/projected/42e4403b-fc10-4515-9021-521d035885c1-kube-api-access-2bjvg\") pod \"cert-manager-operator-controller-manager-55c444bf86-prcfh\" (UID: \"42e4403b-fc10-4515-9021-521d035885c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" Jul 06 19:27:42.475470 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.475434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42e4403b-fc10-4515-9021-521d035885c1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-prcfh\" (UID: \"42e4403b-fc10-4515-9021-521d035885c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" Jul 06 19:27:42.475643 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.475491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bjvg\" (UniqueName: \"kubernetes.io/projected/42e4403b-fc10-4515-9021-521d035885c1-kube-api-access-2bjvg\") pod \"cert-manager-operator-controller-manager-55c444bf86-prcfh\" (UID: \"42e4403b-fc10-4515-9021-521d035885c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" Jul 06 19:27:42.475830 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.475811 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42e4403b-fc10-4515-9021-521d035885c1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-prcfh\" (UID: \"42e4403b-fc10-4515-9021-521d035885c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" Jul 06 19:27:42.484502 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.484469 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bjvg\" (UniqueName: \"kubernetes.io/projected/42e4403b-fc10-4515-9021-521d035885c1-kube-api-access-2bjvg\") pod \"cert-manager-operator-controller-manager-55c444bf86-prcfh\" (UID: \"42e4403b-fc10-4515-9021-521d035885c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" Jul 06 19:27:42.637764 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.637734 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" Jul 06 19:27:42.770164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:42.770142 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh"] Jul 06 19:27:42.772105 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:27:42.772080 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e4403b_fc10_4515_9021_521d035885c1.slice/crio-a9eda57c43c6f99c2f5383bd847cdc3df15db750c395a89f80d26cd54d87a99d WatchSource:0}: Error finding container a9eda57c43c6f99c2f5383bd847cdc3df15db750c395a89f80d26cd54d87a99d: Status 404 returned error can't find the container with id a9eda57c43c6f99c2f5383bd847cdc3df15db750c395a89f80d26cd54d87a99d Jul 06 19:27:43.530142 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:43.530102 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" event={"ID":"42e4403b-fc10-4515-9021-521d035885c1","Type":"ContainerStarted","Data":"a9eda57c43c6f99c2f5383bd847cdc3df15db750c395a89f80d26cd54d87a99d"} Jul 06 19:27:44.535798 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:44.535763 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" event={"ID":"42e4403b-fc10-4515-9021-521d035885c1","Type":"ContainerStarted","Data":"46c341a55dcb9b1d25309a012c1744cf4259d739518716a127380ecdda07acf2"} Jul 06 19:27:44.565237 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:44.565143 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-prcfh" podStartSLOduration=1.001465699 podStartE2EDuration="2.56512211s" totalImagesPullingTime="1.563656411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:44.562035566 +0000 UTC m=+288.659263319" watchObservedRunningTime="2026-07-06 19:27:44.56512211 +0000 UTC m=+288.662349950" Jul 06 19:27:45.730079 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.730043 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9"] Jul 06 19:27:45.732349 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.732332 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:45.735084 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.735057 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:27:45.735240 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.735087 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:27:45.735240 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.735096 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p5qdc\"" Jul 06 19:27:45.745992 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.745965 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9"] Jul 06 19:27:45.798680 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.798641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlk2\" (UniqueName: \"kubernetes.io/projected/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-kube-api-access-bdlk2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:45.798680 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.798680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:45.798922 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.798791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:45.900029 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.899989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlk2\" (UniqueName: \"kubernetes.io/projected/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-kube-api-access-bdlk2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:45.900029 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.900031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:45.900319 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.900072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:45.900443 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.900426 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:45.900518 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.900495 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:45.908994 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:45.908964 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlk2\" (UniqueName: \"kubernetes.io/projected/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-kube-api-access-bdlk2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:46.041670 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:46.041587 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:46.169349 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:46.169287 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9"] Jul 06 19:27:46.171824 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:27:46.171800 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb6ed88_b845_4895_9f0e_3cbcaa14d8f1.slice/crio-fe38da535d5b56a3e15cecdcc24c63afbc20634c2589ef2aefb6993f5b51604e WatchSource:0}: Error finding container fe38da535d5b56a3e15cecdcc24c63afbc20634c2589ef2aefb6993f5b51604e: Status 404 returned error can't find the container with id fe38da535d5b56a3e15cecdcc24c63afbc20634c2589ef2aefb6993f5b51604e Jul 06 19:27:46.545503 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:46.545472 2573 generic.go:356] "Generic (PLEG): container finished" podID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerID="0bdc6dfab90502a9c979f5424fc5132414236ef544b7a76082f4c29faeacaa5a" exitCode=0 Jul 06 19:27:46.545681 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:46.545528 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" event={"ID":"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1","Type":"ContainerDied","Data":"0bdc6dfab90502a9c979f5424fc5132414236ef544b7a76082f4c29faeacaa5a"} Jul 06 19:27:46.545681 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:46.545565 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" event={"ID":"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1","Type":"ContainerStarted","Data":"fe38da535d5b56a3e15cecdcc24c63afbc20634c2589ef2aefb6993f5b51604e"} Jul 06 19:27:48.500952 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.500923 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj"] Jul 06 19:27:48.503935 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.503919 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" Jul 06 19:27:48.506787 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.506762 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pvldl\"" Jul 06 19:27:48.507490 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.507474 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:27:48.507610 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.507505 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:27:48.511540 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.511520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj"] Jul 06 19:27:48.555731 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.555640 2573 generic.go:356] "Generic (PLEG): container finished" podID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerID="2d0788ffbb5dd601f1a9b488d655d6756216963bc04fa551faa04f071274f3d4" exitCode=0 Jul 06 19:27:48.555731 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.555694 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" event={"ID":"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1","Type":"ContainerDied","Data":"2d0788ffbb5dd601f1a9b488d655d6756216963bc04fa551faa04f071274f3d4"} Jul 06 19:27:48.624165 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.624133 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1d60479-1fd5-403a-b5b4-971ad0f4f8b7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-g8dtj\" (UID: \"c1d60479-1fd5-403a-b5b4-971ad0f4f8b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" Jul 06 19:27:48.624165 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.624169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgbrs\" (UniqueName: \"kubernetes.io/projected/c1d60479-1fd5-403a-b5b4-971ad0f4f8b7-kube-api-access-bgbrs\") pod \"cert-manager-cainjector-559cdd8c87-g8dtj\" (UID: \"c1d60479-1fd5-403a-b5b4-971ad0f4f8b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" Jul 06 19:27:48.724593 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.724552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1d60479-1fd5-403a-b5b4-971ad0f4f8b7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-g8dtj\" (UID: \"c1d60479-1fd5-403a-b5b4-971ad0f4f8b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" Jul 06 19:27:48.724593 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.724597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgbrs\" (UniqueName: \"kubernetes.io/projected/c1d60479-1fd5-403a-b5b4-971ad0f4f8b7-kube-api-access-bgbrs\") pod \"cert-manager-cainjector-559cdd8c87-g8dtj\" (UID: \"c1d60479-1fd5-403a-b5b4-971ad0f4f8b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" Jul 06 19:27:48.733735 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.733704 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1d60479-1fd5-403a-b5b4-971ad0f4f8b7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-g8dtj\" (UID: \"c1d60479-1fd5-403a-b5b4-971ad0f4f8b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" Jul 06 19:27:48.733888 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.733866 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgbrs\" (UniqueName: \"kubernetes.io/projected/c1d60479-1fd5-403a-b5b4-971ad0f4f8b7-kube-api-access-bgbrs\") pod \"cert-manager-cainjector-559cdd8c87-g8dtj\" (UID: \"c1d60479-1fd5-403a-b5b4-971ad0f4f8b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" Jul 06 19:27:48.825659 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.825628 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" Jul 06 19:27:48.954978 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:48.954952 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj"] Jul 06 19:27:48.957095 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:27:48.957061 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1d60479_1fd5_403a_b5b4_971ad0f4f8b7.slice/crio-2216c2a7303be18efef52b040f92d3d3786a867a613e00387ae22d0063b21c64 WatchSource:0}: Error finding container 2216c2a7303be18efef52b040f92d3d3786a867a613e00387ae22d0063b21c64: Status 404 returned error can't find the container with id 2216c2a7303be18efef52b040f92d3d3786a867a613e00387ae22d0063b21c64 Jul 06 19:27:49.561670 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:49.561633 2573 generic.go:356] "Generic (PLEG): container finished" podID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerID="d47102ca066e1f8911f1cf8f8bc41b9b15fc81b12810fad1766ed9fe66c6e54f" exitCode=0 Jul 06 19:27:49.562052 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:49.561722 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" event={"ID":"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1","Type":"ContainerDied","Data":"d47102ca066e1f8911f1cf8f8bc41b9b15fc81b12810fad1766ed9fe66c6e54f"} Jul 06 19:27:49.562777 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:49.562759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" event={"ID":"c1d60479-1fd5-403a-b5b4-971ad0f4f8b7","Type":"ContainerStarted","Data":"2216c2a7303be18efef52b040f92d3d3786a867a613e00387ae22d0063b21c64"} Jul 06 19:27:50.717128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.717104 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:50.848263 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.848189 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-bundle\") pod \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " Jul 06 19:27:50.848425 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.848324 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdlk2\" (UniqueName: \"kubernetes.io/projected/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-kube-api-access-bdlk2\") pod \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " Jul 06 19:27:50.848425 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.848372 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-util\") pod \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\" (UID: \"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1\") " Jul 06 19:27:50.848707 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.848644 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-bundle" (OuterVolumeSpecName: "bundle") pod "cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" (UID: "cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:27:50.850843 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.850814 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-kube-api-access-bdlk2" (OuterVolumeSpecName: "kube-api-access-bdlk2") pod "cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" (UID: "cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1"). InnerVolumeSpecName "kube-api-access-bdlk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:27:50.855102 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.855073 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-util" (OuterVolumeSpecName: "util") pod "cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" (UID: "cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:27:50.949130 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.949093 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:50.949130 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.949123 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdlk2\" (UniqueName: \"kubernetes.io/projected/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-kube-api-access-bdlk2\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:50.949130 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:50.949132 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1-util\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:27:51.573496 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:51.573457 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" event={"ID":"cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1","Type":"ContainerDied","Data":"fe38da535d5b56a3e15cecdcc24c63afbc20634c2589ef2aefb6993f5b51604e"} Jul 06 19:27:51.573496 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:51.573499 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp7v9" Jul 06 19:27:51.573725 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:51.573522 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe38da535d5b56a3e15cecdcc24c63afbc20634c2589ef2aefb6993f5b51604e" Jul 06 19:27:52.579304 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.579271 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" event={"ID":"c1d60479-1fd5-403a-b5b4-971ad0f4f8b7","Type":"ContainerStarted","Data":"64fcc00c8d5f10acfe72bf8d79757243aedee5058db0ae9a87f7eade479f098d"} Jul 06 19:27:52.597150 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.597087 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-g8dtj" podStartSLOduration=1.42806418 podStartE2EDuration="4.597065206s" totalImagesPullingTime="3.169001026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:52.595238505 +0000 UTC m=+296.692466262" watchObservedRunningTime="2026-07-06 19:27:52.597065206 +0000 UTC m=+296.694292977" Jul 06 19:27:52.916728 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.916643 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-trpvq"] Jul 06 19:27:52.916949 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.916923 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerName="util" Jul 06 19:27:52.916949 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.916933 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerName="util" Jul 06 19:27:52.916949 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.916948 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerName="pull" Jul 06 19:27:52.917046 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.916953 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerName="pull" Jul 06 19:27:52.917046 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.916961 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerName="extract" Jul 06 19:27:52.917046 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.916968 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerName="extract" Jul 06 19:27:52.917046 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.917026 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb6ed88-b845-4895-9f0e-3cbcaa14d8f1" containerName="extract" Jul 06 19:27:52.919501 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.919485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:27:52.921684 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.921663 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-glb6s\"" Jul 06 19:27:52.928866 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:52.928843 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-trpvq"] Jul 06 19:27:53.068401 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.068365 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khttm\" (UniqueName: \"kubernetes.io/projected/1ee4ef9d-78a0-4412-adba-807cd38b4b4f-kube-api-access-khttm\") pod \"cert-manager-webhook-6486ffd997-trpvq\" (UID: \"1ee4ef9d-78a0-4412-adba-807cd38b4b4f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:27:53.068602 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.068433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ee4ef9d-78a0-4412-adba-807cd38b4b4f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-trpvq\" (UID: \"1ee4ef9d-78a0-4412-adba-807cd38b4b4f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:27:53.169045 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.168951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ee4ef9d-78a0-4412-adba-807cd38b4b4f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-trpvq\" (UID: \"1ee4ef9d-78a0-4412-adba-807cd38b4b4f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:27:53.169243 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.169057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khttm\" (UniqueName: \"kubernetes.io/projected/1ee4ef9d-78a0-4412-adba-807cd38b4b4f-kube-api-access-khttm\") pod \"cert-manager-webhook-6486ffd997-trpvq\" (UID: \"1ee4ef9d-78a0-4412-adba-807cd38b4b4f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:27:53.180747 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.180720 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ee4ef9d-78a0-4412-adba-807cd38b4b4f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-trpvq\" (UID: \"1ee4ef9d-78a0-4412-adba-807cd38b4b4f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:27:53.180864 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.180835 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khttm\" (UniqueName: \"kubernetes.io/projected/1ee4ef9d-78a0-4412-adba-807cd38b4b4f-kube-api-access-khttm\") pod \"cert-manager-webhook-6486ffd997-trpvq\" (UID: \"1ee4ef9d-78a0-4412-adba-807cd38b4b4f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:27:53.228427 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.228398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:27:53.366320 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.366270 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-trpvq"] Jul 06 19:27:53.368898 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:27:53.368865 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ee4ef9d_78a0_4412_adba_807cd38b4b4f.slice/crio-19f3dd34792f4731932e60155b293b5d4bf72030e30194b0df2b6798cb54e522 WatchSource:0}: Error finding container 19f3dd34792f4731932e60155b293b5d4bf72030e30194b0df2b6798cb54e522: Status 404 returned error can't find the container with id 19f3dd34792f4731932e60155b293b5d4bf72030e30194b0df2b6798cb54e522 Jul 06 19:27:53.585581 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.585542 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" event={"ID":"1ee4ef9d-78a0-4412-adba-807cd38b4b4f","Type":"ContainerStarted","Data":"08a650e1142767875e477258aea8b2b5f0f798f4b4b24fef6913a269f72dbea8"} Jul 06 19:27:53.585581 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.585584 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" event={"ID":"1ee4ef9d-78a0-4412-adba-807cd38b4b4f","Type":"ContainerStarted","Data":"19f3dd34792f4731932e60155b293b5d4bf72030e30194b0df2b6798cb54e522"} Jul 06 19:27:53.586051 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.585622 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:27:53.609907 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:53.609855 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" podStartSLOduration=1.6098385990000001 podStartE2EDuration="1.609838599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:53.608009663 +0000 UTC m=+297.705237416" watchObservedRunningTime="2026-07-06 19:27:53.609838599 +0000 UTC m=+297.707066352" Jul 06 19:27:56.272267 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:56.272230 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:27:59.592137 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:27:59.592110 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-trpvq" Jul 06 19:28:01.804812 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:01.804775 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g"] Jul 06 19:28:01.808063 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:01.808044 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:01.810322 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:01.810303 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:28:01.810838 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:01.810807 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:28:01.810838 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:01.810821 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p5qdc\"" Jul 06 19:28:01.817034 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:01.817014 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g"] Jul 06 19:28:01.940585 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:01.940550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:01.940585 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:01.940584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:01.940798 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:01.940709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbr7\" (UniqueName: \"kubernetes.io/projected/047740ac-b0d2-4758-94d6-b79d72e8e533-kube-api-access-tmbr7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:02.041961 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.041927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbr7\" (UniqueName: \"kubernetes.io/projected/047740ac-b0d2-4758-94d6-b79d72e8e533-kube-api-access-tmbr7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:02.041961 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.041965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:02.042192 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.042007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:02.042418 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.042396 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:02.042481 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.042431 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:02.050378 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.050352 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbr7\" (UniqueName: \"kubernetes.io/projected/047740ac-b0d2-4758-94d6-b79d72e8e533-kube-api-access-tmbr7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:02.116978 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.116942 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:02.245438 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.245377 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g"] Jul 06 19:28:02.247948 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:28:02.247919 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047740ac_b0d2_4758_94d6_b79d72e8e533.slice/crio-346272da32176190d45d1243c2b3538622f05c728d5a2f1d42bfdc520ee31813 WatchSource:0}: Error finding container 346272da32176190d45d1243c2b3538622f05c728d5a2f1d42bfdc520ee31813: Status 404 returned error can't find the container with id 346272da32176190d45d1243c2b3538622f05c728d5a2f1d42bfdc520ee31813 Jul 06 19:28:02.619706 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.619675 2573 generic.go:356] "Generic (PLEG): container finished" podID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerID="e6c6911e8694e1a3648e06f32ee1477980d5c4654d1be234ddeae118b1c84da1" exitCode=0 Jul 06 19:28:02.619888 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.619711 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" event={"ID":"047740ac-b0d2-4758-94d6-b79d72e8e533","Type":"ContainerDied","Data":"e6c6911e8694e1a3648e06f32ee1477980d5c4654d1be234ddeae118b1c84da1"} Jul 06 19:28:02.619888 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.619757 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" event={"ID":"047740ac-b0d2-4758-94d6-b79d72e8e533","Type":"ContainerStarted","Data":"346272da32176190d45d1243c2b3538622f05c728d5a2f1d42bfdc520ee31813"} Jul 06 19:28:02.620193 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:02.620181 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:28:03.625899 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:03.625866 2573 generic.go:356] "Generic (PLEG): container finished" podID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerID="2c2d99916779714286d9415fa43ffd583e716aff676c8d8f2a9c3b91fd5f4b2d" exitCode=0 Jul 06 19:28:03.626301 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:03.625958 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" event={"ID":"047740ac-b0d2-4758-94d6-b79d72e8e533","Type":"ContainerDied","Data":"2c2d99916779714286d9415fa43ffd583e716aff676c8d8f2a9c3b91fd5f4b2d"} Jul 06 19:28:04.632025 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:04.631989 2573 generic.go:356] "Generic (PLEG): container finished" podID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerID="bb22cc34eddca970c5b34212b7b0d400419f98d957a4b9e3af60dfce736df444" exitCode=0 Jul 06 19:28:04.632432 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:04.632057 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" event={"ID":"047740ac-b0d2-4758-94d6-b79d72e8e533","Type":"ContainerDied","Data":"bb22cc34eddca970c5b34212b7b0d400419f98d957a4b9e3af60dfce736df444"} Jul 06 19:28:05.779489 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.779459 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:05.877080 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.877051 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-util\") pod \"047740ac-b0d2-4758-94d6-b79d72e8e533\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " Jul 06 19:28:05.877282 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.877093 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmbr7\" (UniqueName: \"kubernetes.io/projected/047740ac-b0d2-4758-94d6-b79d72e8e533-kube-api-access-tmbr7\") pod \"047740ac-b0d2-4758-94d6-b79d72e8e533\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " Jul 06 19:28:05.877282 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.877116 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-bundle\") pod \"047740ac-b0d2-4758-94d6-b79d72e8e533\" (UID: \"047740ac-b0d2-4758-94d6-b79d72e8e533\") " Jul 06 19:28:05.877803 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.877770 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-bundle" (OuterVolumeSpecName: "bundle") pod "047740ac-b0d2-4758-94d6-b79d72e8e533" (UID: "047740ac-b0d2-4758-94d6-b79d72e8e533"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:28:05.879128 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.879105 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047740ac-b0d2-4758-94d6-b79d72e8e533-kube-api-access-tmbr7" (OuterVolumeSpecName: "kube-api-access-tmbr7") pod "047740ac-b0d2-4758-94d6-b79d72e8e533" (UID: "047740ac-b0d2-4758-94d6-b79d72e8e533"). InnerVolumeSpecName "kube-api-access-tmbr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:28:05.882606 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.882568 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-util" (OuterVolumeSpecName: "util") pod "047740ac-b0d2-4758-94d6-b79d72e8e533" (UID: "047740ac-b0d2-4758-94d6-b79d72e8e533"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:28:05.978713 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.978621 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmbr7\" (UniqueName: \"kubernetes.io/projected/047740ac-b0d2-4758-94d6-b79d72e8e533-kube-api-access-tmbr7\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:28:05.978713 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.978655 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:28:05.978713 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:05.978667 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/047740ac-b0d2-4758-94d6-b79d72e8e533-util\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:28:06.641989 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.641943 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" event={"ID":"047740ac-b0d2-4758-94d6-b79d72e8e533","Type":"ContainerDied","Data":"346272da32176190d45d1243c2b3538622f05c728d5a2f1d42bfdc520ee31813"} Jul 06 19:28:06.641989 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.641985 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5t94g" Jul 06 19:28:06.642191 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.642005 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346272da32176190d45d1243c2b3538622f05c728d5a2f1d42bfdc520ee31813" Jul 06 19:28:06.768866 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.768778 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-45m6h"] Jul 06 19:28:06.769164 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.769147 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerName="util" Jul 06 19:28:06.769281 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.769166 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerName="util" Jul 06 19:28:06.769281 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.769190 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerName="pull" Jul 06 19:28:06.769281 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.769199 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerName="pull" Jul 06 19:28:06.769281 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.769224 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerName="extract" Jul 06 19:28:06.769281 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.769233 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerName="extract" Jul 06 19:28:06.769503 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.769312 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="047740ac-b0d2-4758-94d6-b79d72e8e533" containerName="extract" Jul 06 19:28:06.792684 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.792653 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-45m6h"] Jul 06 19:28:06.793108 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.792785 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-45m6h" Jul 06 19:28:06.795025 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.795002 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-k55lx\"" Jul 06 19:28:06.885672 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.885632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpl2g\" (UniqueName: \"kubernetes.io/projected/37d27f0a-77cd-4d8f-85b6-3dbc73a57d54-kube-api-access-kpl2g\") pod \"cert-manager-55b4d954d5-45m6h\" (UID: \"37d27f0a-77cd-4d8f-85b6-3dbc73a57d54\") " pod="cert-manager/cert-manager-55b4d954d5-45m6h" Jul 06 19:28:06.885672 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.885680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37d27f0a-77cd-4d8f-85b6-3dbc73a57d54-bound-sa-token\") pod \"cert-manager-55b4d954d5-45m6h\" (UID: \"37d27f0a-77cd-4d8f-85b6-3dbc73a57d54\") " pod="cert-manager/cert-manager-55b4d954d5-45m6h" Jul 06 19:28:06.986111 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.986084 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpl2g\" (UniqueName: \"kubernetes.io/projected/37d27f0a-77cd-4d8f-85b6-3dbc73a57d54-kube-api-access-kpl2g\") pod \"cert-manager-55b4d954d5-45m6h\" (UID: \"37d27f0a-77cd-4d8f-85b6-3dbc73a57d54\") " pod="cert-manager/cert-manager-55b4d954d5-45m6h" Jul 06 19:28:06.986111 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.986118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37d27f0a-77cd-4d8f-85b6-3dbc73a57d54-bound-sa-token\") pod \"cert-manager-55b4d954d5-45m6h\" (UID: \"37d27f0a-77cd-4d8f-85b6-3dbc73a57d54\") " pod="cert-manager/cert-manager-55b4d954d5-45m6h" Jul 06 19:28:06.995064 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.995030 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37d27f0a-77cd-4d8f-85b6-3dbc73a57d54-bound-sa-token\") pod \"cert-manager-55b4d954d5-45m6h\" (UID: \"37d27f0a-77cd-4d8f-85b6-3dbc73a57d54\") " pod="cert-manager/cert-manager-55b4d954d5-45m6h" Jul 06 19:28:06.995064 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:06.995053 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpl2g\" (UniqueName: \"kubernetes.io/projected/37d27f0a-77cd-4d8f-85b6-3dbc73a57d54-kube-api-access-kpl2g\") pod \"cert-manager-55b4d954d5-45m6h\" (UID: \"37d27f0a-77cd-4d8f-85b6-3dbc73a57d54\") " pod="cert-manager/cert-manager-55b4d954d5-45m6h" Jul 06 19:28:07.102052 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:07.102021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-45m6h" Jul 06 19:28:07.258964 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:07.258908 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-45m6h"] Jul 06 19:28:07.263230 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:28:07.263183 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37d27f0a_77cd_4d8f_85b6_3dbc73a57d54.slice/crio-fa37bf609b271dc5ab0d54f6239e856d3362a34565712e58b115ef6667847d30 WatchSource:0}: Error finding container fa37bf609b271dc5ab0d54f6239e856d3362a34565712e58b115ef6667847d30: Status 404 returned error can't find the container with id fa37bf609b271dc5ab0d54f6239e856d3362a34565712e58b115ef6667847d30 Jul 06 19:28:07.647718 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:07.647627 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-45m6h" event={"ID":"37d27f0a-77cd-4d8f-85b6-3dbc73a57d54","Type":"ContainerStarted","Data":"bcbde82d95f80b04b7694de685fbd306f8efbbeee34cbdf7aec69645559ef178"} Jul 06 19:28:07.647718 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:07.647669 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-45m6h" event={"ID":"37d27f0a-77cd-4d8f-85b6-3dbc73a57d54","Type":"ContainerStarted","Data":"fa37bf609b271dc5ab0d54f6239e856d3362a34565712e58b115ef6667847d30"} Jul 06 19:28:07.669259 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:07.669190 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-45m6h" podStartSLOduration=1.669172358 podStartE2EDuration="1.669172358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:07.668220973 +0000 UTC m=+311.765448723" watchObservedRunningTime="2026-07-06 19:28:07.669172358 +0000 UTC m=+311.766400112" Jul 06 19:28:18.767465 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.767430 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg"] Jul 06 19:28:18.769847 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.769832 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.776977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.776955 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:28:18.777114 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.777004 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:28:18.777114 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.777017 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:28:18.777234 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.777145 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zpx5b\"" Jul 06 19:28:18.777298 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.777286 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:28:18.783538 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.783515 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87de029c-e31a-42a6-afc3-476eb4834a9c-tmp\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.783633 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.783562 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87de029c-e31a-42a6-afc3-476eb4834a9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.783633 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.783622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4ql\" (UniqueName: \"kubernetes.io/projected/87de029c-e31a-42a6-afc3-476eb4834a9c-kube-api-access-nb4ql\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.783700 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.783655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/87de029c-e31a-42a6-afc3-476eb4834a9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.783700 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.783675 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/87de029c-e31a-42a6-afc3-476eb4834a9c-manifests\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.856392 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.856355 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg"] Jul 06 19:28:18.884714 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.884673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4ql\" (UniqueName: \"kubernetes.io/projected/87de029c-e31a-42a6-afc3-476eb4834a9c-kube-api-access-nb4ql\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.884895 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.884718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/87de029c-e31a-42a6-afc3-476eb4834a9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.884895 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.884741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/87de029c-e31a-42a6-afc3-476eb4834a9c-manifests\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.884895 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.884770 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87de029c-e31a-42a6-afc3-476eb4834a9c-tmp\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.884895 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.884810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87de029c-e31a-42a6-afc3-476eb4834a9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.885172 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.885144 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/87de029c-e31a-42a6-afc3-476eb4834a9c-manifests\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.887060 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.887031 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87de029c-e31a-42a6-afc3-476eb4834a9c-tmp\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.887277 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.887260 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87de029c-e31a-42a6-afc3-476eb4834a9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.887384 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.887366 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/87de029c-e31a-42a6-afc3-476eb4834a9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:18.895950 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:18.895930 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4ql\" (UniqueName: \"kubernetes.io/projected/87de029c-e31a-42a6-afc3-476eb4834a9c-kube-api-access-nb4ql\") pod \"opendatahub-operator-controller-manager-7865995f8c-q6vmg\" (UID: \"87de029c-e31a-42a6-afc3-476eb4834a9c\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:19.080583 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:19.080557 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:19.250988 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:19.250960 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg"] Jul 06 19:28:19.253113 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:28:19.253079 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87de029c_e31a_42a6_afc3_476eb4834a9c.slice/crio-e34e9237fa00244c045777d462920cd10b6786b6275273b5c56455ad2f027c8b WatchSource:0}: Error finding container e34e9237fa00244c045777d462920cd10b6786b6275273b5c56455ad2f027c8b: Status 404 returned error can't find the container with id e34e9237fa00244c045777d462920cd10b6786b6275273b5c56455ad2f027c8b Jul 06 19:28:19.699896 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:19.699859 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" event={"ID":"87de029c-e31a-42a6-afc3-476eb4834a9c","Type":"ContainerStarted","Data":"e34e9237fa00244c045777d462920cd10b6786b6275273b5c56455ad2f027c8b"} Jul 06 19:28:22.716617 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:22.716576 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" event={"ID":"87de029c-e31a-42a6-afc3-476eb4834a9c","Type":"ContainerStarted","Data":"4216d1eea632016fbe6f06ce9439124ab170ca0a43e01ef89ec50f86a26e6d6b"} Jul 06 19:28:23.722569 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:23.722531 2573 generic.go:356] "Generic (PLEG): container finished" podID="87de029c-e31a-42a6-afc3-476eb4834a9c" containerID="4216d1eea632016fbe6f06ce9439124ab170ca0a43e01ef89ec50f86a26e6d6b" exitCode=0 Jul 06 19:28:23.722969 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:23.722621 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" event={"ID":"87de029c-e31a-42a6-afc3-476eb4834a9c","Type":"ContainerDied","Data":"4216d1eea632016fbe6f06ce9439124ab170ca0a43e01ef89ec50f86a26e6d6b"} Jul 06 19:28:24.728708 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:24.728674 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" event={"ID":"87de029c-e31a-42a6-afc3-476eb4834a9c","Type":"ContainerStarted","Data":"486e1a27613ca4e1755f2ef0749af8186eb1bf1df9122cc2bcaf8146f75726ca"} Jul 06 19:28:24.729089 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:24.728770 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:24.758182 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:24.758113 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" podStartSLOduration=2.454774457 podStartE2EDuration="6.758071006s" totalImagesPullingTime="3.293169864s" totalInitContainerRuntime="1.010126685s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:24.757254297 +0000 UTC m=+328.854482051" watchObservedRunningTime="2026-07-06 19:28:24.758071006 +0000 UTC m=+328.855298759" Jul 06 19:28:35.736034 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:35.735998 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-q6vmg" Jul 06 19:28:39.355709 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.355670 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq"] Jul 06 19:28:39.363820 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.363803 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.395869 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.395838 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq"] Jul 06 19:28:39.400831 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.400808 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z5dz6\"" Jul 06 19:28:39.428590 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.428554 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:28:39.428758 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.428722 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:28:39.432246 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.432222 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:28:39.432668 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.432653 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:28:39.432758 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.432658 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:28:39.458449 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.458421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-metrics-cert\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.458615 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.458471 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnwq\" (UniqueName: \"kubernetes.io/projected/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-kube-api-access-8cnwq\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.458615 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.458497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-manager-config\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.458615 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.458547 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-cert\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.559283 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.559247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-metrics-cert\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.559487 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.559293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnwq\" (UniqueName: \"kubernetes.io/projected/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-kube-api-access-8cnwq\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.559487 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.559315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-manager-config\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.559487 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.559356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-cert\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.560024 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.560002 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-manager-config\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.561783 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.561761 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-metrics-cert\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.561859 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.561820 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-cert\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.577594 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.577566 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnwq\" (UniqueName: \"kubernetes.io/projected/e17f75ab-ddd9-4fe5-8a4b-b8f969a45874-kube-api-access-8cnwq\") pod \"lws-controller-manager-54d6cd8878-9x5fq\" (UID: \"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\") " pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.673450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.673357 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:39.883175 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:39.883049 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq"] Jul 06 19:28:39.885366 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:28:39.885331 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17f75ab_ddd9_4fe5_8a4b_b8f969a45874.slice/crio-9ac3ef4b41452a2352dfe1c807d7275ece05700d066cbc9b76def2da667a2059 WatchSource:0}: Error finding container 9ac3ef4b41452a2352dfe1c807d7275ece05700d066cbc9b76def2da667a2059: Status 404 returned error can't find the container with id 9ac3ef4b41452a2352dfe1c807d7275ece05700d066cbc9b76def2da667a2059 Jul 06 19:28:40.799465 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:40.799433 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" event={"ID":"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874","Type":"ContainerStarted","Data":"9ac3ef4b41452a2352dfe1c807d7275ece05700d066cbc9b76def2da667a2059"} Jul 06 19:28:42.810135 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:42.810101 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" event={"ID":"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874","Type":"ContainerStarted","Data":"2fc019726d121432255fb0eab1b36d800623dee08afce4eddb2ab4d1dee1442e"} Jul 06 19:28:42.810507 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:42.810218 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:42.841984 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:42.841928 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" podStartSLOduration=1.630276002 podStartE2EDuration="3.841907853s" totalImagesPullingTime="2.211631851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:42.838366688 +0000 UTC m=+346.935594442" watchObservedRunningTime="2026-07-06 19:28:42.841907853 +0000 UTC m=+346.939135608" Jul 06 19:28:53.501258 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.501141 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk"] Jul 06 19:28:53.505422 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.505401 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.522685 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.522652 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 19:28:53.522685 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.522652 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 19:28:53.522907 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.522730 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 19:28:53.527120 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.527103 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:28:53.534460 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.534442 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w2dkh\"" Jul 06 19:28:53.534460 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.534453 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:28:53.534985 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.534970 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:28:53.583429 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.583391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.583540 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.583456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.583540 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.583511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.583540 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.583532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.583654 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.583600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swm2d\" (UniqueName: \"kubernetes.io/projected/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-kube-api-access-swm2d\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.583654 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.583642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.583724 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.583663 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.638885 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.638850 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk"] Jul 06 19:28:53.684257 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.684196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.684462 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.684304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.684462 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.684338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.684462 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.684376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swm2d\" (UniqueName: \"kubernetes.io/projected/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-kube-api-access-swm2d\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.684462 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.684401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.684462 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.684426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.684462 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.684457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.685153 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.685126 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.686653 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.686622 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.686751 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.686677 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.686968 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.686946 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.687115 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.687097 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.727725 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.727689 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.737509 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.737479 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swm2d\" (UniqueName: \"kubernetes.io/projected/6686c259-eb37-4cba-b4b7-8fb2a04b4b36-kube-api-access-swm2d\") pod \"istiod-openshift-gateway-94bb8fbfd-fplgk\" (UID: \"6686c259-eb37-4cba-b4b7-8fb2a04b4b36\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.815033 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.814945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:53.816756 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.816734 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:28:53.987089 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:53.987057 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk"] Jul 06 19:28:53.988884 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:28:53.988860 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6686c259_eb37_4cba_b4b7_8fb2a04b4b36.slice/crio-4f4b369d84b6feaecdbc64fc5bb378cb2799bd36da8e543e05ef113d6eaa93ad WatchSource:0}: Error finding container 4f4b369d84b6feaecdbc64fc5bb378cb2799bd36da8e543e05ef113d6eaa93ad: Status 404 returned error can't find the container with id 4f4b369d84b6feaecdbc64fc5bb378cb2799bd36da8e543e05ef113d6eaa93ad Jul 06 19:28:54.864685 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:54.864635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" event={"ID":"6686c259-eb37-4cba-b4b7-8fb2a04b4b36","Type":"ContainerStarted","Data":"4f4b369d84b6feaecdbc64fc5bb378cb2799bd36da8e543e05ef113d6eaa93ad"} Jul 06 19:28:56.880584 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:56.880544 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:28:56.880841 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:56.880617 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:28:57.880833 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:57.880792 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" event={"ID":"6686c259-eb37-4cba-b4b7-8fb2a04b4b36","Type":"ContainerStarted","Data":"cd0874b74f27cb896556a4a6ceece84eda3914ca449061413d0d2ca3827e5dfc"} Jul 06 19:28:57.881318 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:57.880959 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:28:57.886676 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:57.886639 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-fplgk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:28:57.886844 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:57.886731 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" podUID="6686c259-eb37-4cba-b4b7-8fb2a04b4b36" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:28:58.886256 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:58.886197 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-fplgk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:28:58.886613 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:28:58.886279 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" podUID="6686c259-eb37-4cba-b4b7-8fb2a04b4b36" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:29:01.886903 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:01.886868 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" Jul 06 19:29:01.924910 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:01.924853 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fplgk" podStartSLOduration=6.035485464 podStartE2EDuration="8.924834992s" totalImagesPullingTime="2.889349528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:58.287590324 +0000 UTC m=+362.384818079" watchObservedRunningTime="2026-07-06 19:29:01.924834992 +0000 UTC m=+366.022062747" Jul 06 19:29:53.821552 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:53.821521 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9"] Jul 06 19:29:53.825254 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:53.825233 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:53.828951 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:53.828924 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mg4xx\"" Jul 06 19:29:53.829072 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:53.828971 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:29:53.829072 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:53.828971 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:29:53.841254 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:53.841230 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9"] Jul 06 19:29:53.991090 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:53.991046 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:53.991289 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:53.991099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rn4t\" (UniqueName: \"kubernetes.io/projected/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-kube-api-access-2rn4t\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:53.991289 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:53.991152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:54.091924 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.091889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:54.092108 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.091944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rn4t\" (UniqueName: \"kubernetes.io/projected/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-kube-api-access-2rn4t\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:54.092108 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.091974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:54.092411 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.092371 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:54.092488 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.092392 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:54.106169 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.106142 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rn4t\" (UniqueName: \"kubernetes.io/projected/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-kube-api-access-2rn4t\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:54.134070 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.134044 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:54.177709 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.177661 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc"] Jul 06 19:29:54.181732 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.181698 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.194040 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.194006 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc"] Jul 06 19:29:54.273695 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.273664 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9"] Jul 06 19:29:54.275520 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:29:54.275494 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1523ea4a_cced_42ce_b7d2_532ec4baf1f5.slice/crio-680fdb35880c7871badafbd982d05c14b756cdec2f45343fc294c87f7af6bad4 WatchSource:0}: Error finding container 680fdb35880c7871badafbd982d05c14b756cdec2f45343fc294c87f7af6bad4: Status 404 returned error can't find the container with id 680fdb35880c7871badafbd982d05c14b756cdec2f45343fc294c87f7af6bad4 Jul 06 19:29:54.294154 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.294129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.294295 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.294176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.294366 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.294287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjczl\" (UniqueName: \"kubernetes.io/projected/0470f7d5-5f99-4019-9eea-336475adb661-kube-api-access-pjczl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.394968 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.394941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.395101 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.394996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjczl\" (UniqueName: \"kubernetes.io/projected/0470f7d5-5f99-4019-9eea-336475adb661-kube-api-access-pjczl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.395101 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.395030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.395418 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.395400 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.395467 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.395407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.408614 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.408589 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjczl\" (UniqueName: \"kubernetes.io/projected/0470f7d5-5f99-4019-9eea-336475adb661-kube-api-access-pjczl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.495140 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.495104 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:54.578654 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.578486 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74"] Jul 06 19:29:54.582316 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.582294 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.596633 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.596560 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74"] Jul 06 19:29:54.632073 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.632047 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc"] Jul 06 19:29:54.633379 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:29:54.633351 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0470f7d5_5f99_4019_9eea_336475adb661.slice/crio-90c87dc2c7040f1e94eef4ba144c81307feced33267de32f89ffce870e2f3829 WatchSource:0}: Error finding container 90c87dc2c7040f1e94eef4ba144c81307feced33267de32f89ffce870e2f3829: Status 404 returned error can't find the container with id 90c87dc2c7040f1e94eef4ba144c81307feced33267de32f89ffce870e2f3829 Jul 06 19:29:54.697931 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.697904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.698085 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.697951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.698085 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.698022 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8brc\" (UniqueName: \"kubernetes.io/projected/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-kube-api-access-p8brc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.798512 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.798477 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8brc\" (UniqueName: \"kubernetes.io/projected/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-kube-api-access-p8brc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.798681 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.798555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.798681 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.798574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.798919 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.798903 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.798982 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.798966 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.807242 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.807196 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8brc\" (UniqueName: \"kubernetes.io/projected/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-kube-api-access-p8brc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.892858 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.892746 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:54.978633 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.978605 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns"] Jul 06 19:29:54.982836 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.982810 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:54.994044 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:54.993946 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns"] Jul 06 19:29:55.034485 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.034458 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74"] Jul 06 19:29:55.035957 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:29:55.035932 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05a45cd_a8d4_4e53_9a0e_d373374a70e0.slice/crio-2bf6402bd8523e6501d1414f8050329b46c14e6014e98dce5b15adcd04074041 WatchSource:0}: Error finding container 2bf6402bd8523e6501d1414f8050329b46c14e6014e98dce5b15adcd04074041: Status 404 returned error can't find the container with id 2bf6402bd8523e6501d1414f8050329b46c14e6014e98dce5b15adcd04074041 Jul 06 19:29:55.100876 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.100848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.101033 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.100912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.101033 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.100955 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgz5n\" (UniqueName: \"kubernetes.io/projected/7fbeec71-aed1-4625-9030-2431a05a7145-kube-api-access-sgz5n\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.137954 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.137923 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" event={"ID":"b05a45cd-a8d4-4e53-9a0e-d373374a70e0","Type":"ContainerStarted","Data":"18ff301847d752fe5d62781a86fcaed7cccee55d11d52fec3b2b917779bc79ad"} Jul 06 19:29:55.138072 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.137987 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" event={"ID":"b05a45cd-a8d4-4e53-9a0e-d373374a70e0","Type":"ContainerStarted","Data":"2bf6402bd8523e6501d1414f8050329b46c14e6014e98dce5b15adcd04074041"} Jul 06 19:29:55.139585 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.139561 2573 generic.go:356] "Generic (PLEG): container finished" podID="0470f7d5-5f99-4019-9eea-336475adb661" containerID="ee761d5046a3b8676c12dd715f7c07bac6113906d4e2c51af0d16de34f9ab4d4" exitCode=0 Jul 06 19:29:55.139679 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.139644 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" event={"ID":"0470f7d5-5f99-4019-9eea-336475adb661","Type":"ContainerDied","Data":"ee761d5046a3b8676c12dd715f7c07bac6113906d4e2c51af0d16de34f9ab4d4"} Jul 06 19:29:55.139738 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.139684 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" event={"ID":"0470f7d5-5f99-4019-9eea-336475adb661","Type":"ContainerStarted","Data":"90c87dc2c7040f1e94eef4ba144c81307feced33267de32f89ffce870e2f3829"} Jul 06 19:29:55.141245 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.141223 2573 generic.go:356] "Generic (PLEG): container finished" podID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerID="2b119b98177c15b502b3c9e3b4f88f778b69dc5fea0bf9ae62ca3da3f6b3d4c4" exitCode=0 Jul 06 19:29:55.141328 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.141306 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" event={"ID":"1523ea4a-cced-42ce-b7d2-532ec4baf1f5","Type":"ContainerDied","Data":"2b119b98177c15b502b3c9e3b4f88f778b69dc5fea0bf9ae62ca3da3f6b3d4c4"} Jul 06 19:29:55.141367 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.141355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" event={"ID":"1523ea4a-cced-42ce-b7d2-532ec4baf1f5","Type":"ContainerStarted","Data":"680fdb35880c7871badafbd982d05c14b756cdec2f45343fc294c87f7af6bad4"} Jul 06 19:29:55.202069 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.202020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.202069 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.202072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgz5n\" (UniqueName: \"kubernetes.io/projected/7fbeec71-aed1-4625-9030-2431a05a7145-kube-api-access-sgz5n\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.202342 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.202131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.202501 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.202480 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.202501 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.202495 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.215399 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.215371 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgz5n\" (UniqueName: \"kubernetes.io/projected/7fbeec71-aed1-4625-9030-2431a05a7145-kube-api-access-sgz5n\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.295571 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.295539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:29:55.433393 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:55.433322 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns"] Jul 06 19:29:55.435834 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:29:55.435814 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbeec71_aed1_4625_9030_2431a05a7145.slice/crio-95bb1eb1e8400e3530ab716616454b376578100c7a09512266a8091645608def WatchSource:0}: Error finding container 95bb1eb1e8400e3530ab716616454b376578100c7a09512266a8091645608def: Status 404 returned error can't find the container with id 95bb1eb1e8400e3530ab716616454b376578100c7a09512266a8091645608def Jul 06 19:29:56.148722 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:56.148690 2573 generic.go:356] "Generic (PLEG): container finished" podID="7fbeec71-aed1-4625-9030-2431a05a7145" containerID="5befd5623427186b51cc80fe2ceffc0a0d1555497be9d20a82a476a4e29ae6be" exitCode=0 Jul 06 19:29:56.149159 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:56.148768 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" event={"ID":"7fbeec71-aed1-4625-9030-2431a05a7145","Type":"ContainerDied","Data":"5befd5623427186b51cc80fe2ceffc0a0d1555497be9d20a82a476a4e29ae6be"} Jul 06 19:29:56.149159 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:56.148827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" event={"ID":"7fbeec71-aed1-4625-9030-2431a05a7145","Type":"ContainerStarted","Data":"95bb1eb1e8400e3530ab716616454b376578100c7a09512266a8091645608def"} Jul 06 19:29:56.150555 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:56.150534 2573 generic.go:356] "Generic (PLEG): container finished" podID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerID="18ff301847d752fe5d62781a86fcaed7cccee55d11d52fec3b2b917779bc79ad" exitCode=0 Jul 06 19:29:56.150670 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:56.150608 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" event={"ID":"b05a45cd-a8d4-4e53-9a0e-d373374a70e0","Type":"ContainerDied","Data":"18ff301847d752fe5d62781a86fcaed7cccee55d11d52fec3b2b917779bc79ad"} Jul 06 19:29:56.152500 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:56.152473 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" event={"ID":"0470f7d5-5f99-4019-9eea-336475adb661","Type":"ContainerStarted","Data":"b8ac781866832ccc901cdee61b031dace88a2fdb28c56da9bbe9e773f3502ab3"} Jul 06 19:29:56.154387 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:56.154354 2573 generic.go:356] "Generic (PLEG): container finished" podID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerID="31c29cd515765b97d5be84c46fc9a39bd3fc14131edb129c029c935b1089bc3c" exitCode=0 Jul 06 19:29:56.154534 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:56.154429 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" event={"ID":"1523ea4a-cced-42ce-b7d2-532ec4baf1f5","Type":"ContainerDied","Data":"31c29cd515765b97d5be84c46fc9a39bd3fc14131edb129c029c935b1089bc3c"} Jul 06 19:29:57.160844 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:57.160750 2573 generic.go:356] "Generic (PLEG): container finished" podID="7fbeec71-aed1-4625-9030-2431a05a7145" containerID="72c46e72e0f2ca4d50010666f383801f9c503a276a6223f955eb13fc5d59e791" exitCode=0 Jul 06 19:29:57.161351 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:57.160846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" event={"ID":"7fbeec71-aed1-4625-9030-2431a05a7145","Type":"ContainerDied","Data":"72c46e72e0f2ca4d50010666f383801f9c503a276a6223f955eb13fc5d59e791"} Jul 06 19:29:57.162810 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:57.162790 2573 generic.go:356] "Generic (PLEG): container finished" podID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerID="69ff2910de784ad0b071338e9de988cc40dbd79b3e92787c77f3bfeca7159c30" exitCode=0 Jul 06 19:29:57.162901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:57.162867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" event={"ID":"b05a45cd-a8d4-4e53-9a0e-d373374a70e0","Type":"ContainerDied","Data":"69ff2910de784ad0b071338e9de988cc40dbd79b3e92787c77f3bfeca7159c30"} Jul 06 19:29:57.164815 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:57.164793 2573 generic.go:356] "Generic (PLEG): container finished" podID="0470f7d5-5f99-4019-9eea-336475adb661" containerID="b8ac781866832ccc901cdee61b031dace88a2fdb28c56da9bbe9e773f3502ab3" exitCode=0 Jul 06 19:29:57.164916 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:57.164880 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" event={"ID":"0470f7d5-5f99-4019-9eea-336475adb661","Type":"ContainerDied","Data":"b8ac781866832ccc901cdee61b031dace88a2fdb28c56da9bbe9e773f3502ab3"} Jul 06 19:29:57.169629 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:57.169603 2573 generic.go:356] "Generic (PLEG): container finished" podID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerID="53f5057407580ce30bc58c066558e3fbf9e322d949bb710027662b49633612c9" exitCode=0 Jul 06 19:29:57.170088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:57.170060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" event={"ID":"1523ea4a-cced-42ce-b7d2-532ec4baf1f5","Type":"ContainerDied","Data":"53f5057407580ce30bc58c066558e3fbf9e322d949bb710027662b49633612c9"} Jul 06 19:29:58.179944 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.179906 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" event={"ID":"7fbeec71-aed1-4625-9030-2431a05a7145","Type":"ContainerStarted","Data":"f8f81fd6d58322e826f9233a22182dd589b16e17900d1fa3eb53137e97bc7437"} Jul 06 19:29:58.182012 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.181983 2573 generic.go:356] "Generic (PLEG): container finished" podID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerID="48a2a9379bfb0eab4b17f4dd5c6cd268d5fc2756ee77ffacad6c4bc39d1583cd" exitCode=0 Jul 06 19:29:58.182130 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.182055 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" event={"ID":"b05a45cd-a8d4-4e53-9a0e-d373374a70e0","Type":"ContainerDied","Data":"48a2a9379bfb0eab4b17f4dd5c6cd268d5fc2756ee77ffacad6c4bc39d1583cd"} Jul 06 19:29:58.184004 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.183977 2573 generic.go:356] "Generic (PLEG): container finished" podID="0470f7d5-5f99-4019-9eea-336475adb661" containerID="91b1f08962630d84733830cb060739c690ce3c7c289d464f82223fe29492b397" exitCode=0 Jul 06 19:29:58.184101 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.184058 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" event={"ID":"0470f7d5-5f99-4019-9eea-336475adb661","Type":"ContainerDied","Data":"91b1f08962630d84733830cb060739c690ce3c7c289d464f82223fe29492b397"} Jul 06 19:29:58.202139 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.202087 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" podStartSLOduration=3.492730141 podStartE2EDuration="4.202072922s" totalImagesPullingTime="709.342781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:58.19978925 +0000 UTC m=+422.297017006" watchObservedRunningTime="2026-07-06 19:29:58.202072922 +0000 UTC m=+422.299300676" Jul 06 19:29:58.319388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.319364 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:58.430874 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.430794 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-util\") pod \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " Jul 06 19:29:58.430874 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.430854 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-bundle\") pod \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " Jul 06 19:29:58.431059 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.430887 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rn4t\" (UniqueName: \"kubernetes.io/projected/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-kube-api-access-2rn4t\") pod \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\" (UID: \"1523ea4a-cced-42ce-b7d2-532ec4baf1f5\") " Jul 06 19:29:58.431424 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.431391 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-bundle" (OuterVolumeSpecName: "bundle") pod "1523ea4a-cced-42ce-b7d2-532ec4baf1f5" (UID: "1523ea4a-cced-42ce-b7d2-532ec4baf1f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:29:58.433171 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.433149 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-kube-api-access-2rn4t" (OuterVolumeSpecName: "kube-api-access-2rn4t") pod "1523ea4a-cced-42ce-b7d2-532ec4baf1f5" (UID: "1523ea4a-cced-42ce-b7d2-532ec4baf1f5"). InnerVolumeSpecName "kube-api-access-2rn4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:58.435458 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.435424 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-util" (OuterVolumeSpecName: "util") pod "1523ea4a-cced-42ce-b7d2-532ec4baf1f5" (UID: "1523ea4a-cced-42ce-b7d2-532ec4baf1f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:29:58.532318 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.532267 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-util\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:29:58.532318 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.532310 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:29:58.532318 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:58.532322 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rn4t\" (UniqueName: \"kubernetes.io/projected/1523ea4a-cced-42ce-b7d2-532ec4baf1f5-kube-api-access-2rn4t\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:29:59.190469 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.190439 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" Jul 06 19:29:59.191088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.190476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9" event={"ID":"1523ea4a-cced-42ce-b7d2-532ec4baf1f5","Type":"ContainerDied","Data":"680fdb35880c7871badafbd982d05c14b756cdec2f45343fc294c87f7af6bad4"} Jul 06 19:29:59.191088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.190523 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="680fdb35880c7871badafbd982d05c14b756cdec2f45343fc294c87f7af6bad4" Jul 06 19:29:59.318593 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.318568 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:29:59.351581 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.351557 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:29:59.441349 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.441262 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjczl\" (UniqueName: \"kubernetes.io/projected/0470f7d5-5f99-4019-9eea-336475adb661-kube-api-access-pjczl\") pod \"0470f7d5-5f99-4019-9eea-336475adb661\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " Jul 06 19:29:59.441349 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.441316 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-bundle\") pod \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " Jul 06 19:29:59.441349 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.441348 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8brc\" (UniqueName: \"kubernetes.io/projected/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-kube-api-access-p8brc\") pod \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " Jul 06 19:29:59.441624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.441379 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-bundle\") pod \"0470f7d5-5f99-4019-9eea-336475adb661\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " Jul 06 19:29:59.441624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.441398 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-util\") pod \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\" (UID: \"b05a45cd-a8d4-4e53-9a0e-d373374a70e0\") " Jul 06 19:29:59.441624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.441438 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-util\") pod \"0470f7d5-5f99-4019-9eea-336475adb661\" (UID: \"0470f7d5-5f99-4019-9eea-336475adb661\") " Jul 06 19:29:59.441965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.441925 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-bundle" (OuterVolumeSpecName: "bundle") pod "0470f7d5-5f99-4019-9eea-336475adb661" (UID: "0470f7d5-5f99-4019-9eea-336475adb661"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:29:59.442047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.441949 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-bundle" (OuterVolumeSpecName: "bundle") pod "b05a45cd-a8d4-4e53-9a0e-d373374a70e0" (UID: "b05a45cd-a8d4-4e53-9a0e-d373374a70e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:29:59.443488 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.443463 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-kube-api-access-p8brc" (OuterVolumeSpecName: "kube-api-access-p8brc") pod "b05a45cd-a8d4-4e53-9a0e-d373374a70e0" (UID: "b05a45cd-a8d4-4e53-9a0e-d373374a70e0"). InnerVolumeSpecName "kube-api-access-p8brc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:59.443598 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.443508 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0470f7d5-5f99-4019-9eea-336475adb661-kube-api-access-pjczl" (OuterVolumeSpecName: "kube-api-access-pjczl") pod "0470f7d5-5f99-4019-9eea-336475adb661" (UID: "0470f7d5-5f99-4019-9eea-336475adb661"). InnerVolumeSpecName "kube-api-access-pjczl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:59.446860 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.446823 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-util" (OuterVolumeSpecName: "util") pod "0470f7d5-5f99-4019-9eea-336475adb661" (UID: "0470f7d5-5f99-4019-9eea-336475adb661"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:29:59.449276 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.449250 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-util" (OuterVolumeSpecName: "util") pod "b05a45cd-a8d4-4e53-9a0e-d373374a70e0" (UID: "b05a45cd-a8d4-4e53-9a0e-d373374a70e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:29:59.542759 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.542720 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjczl\" (UniqueName: \"kubernetes.io/projected/0470f7d5-5f99-4019-9eea-336475adb661-kube-api-access-pjczl\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:29:59.542759 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.542750 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:29:59.542759 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.542759 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8brc\" (UniqueName: \"kubernetes.io/projected/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-kube-api-access-p8brc\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:29:59.542759 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.542769 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:29:59.543024 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.542777 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b05a45cd-a8d4-4e53-9a0e-d373374a70e0-util\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:29:59.543024 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:29:59.542785 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0470f7d5-5f99-4019-9eea-336475adb661-util\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:30:00.196758 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:00.196729 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" Jul 06 19:30:00.197200 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:00.196726 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74" event={"ID":"b05a45cd-a8d4-4e53-9a0e-d373374a70e0","Type":"ContainerDied","Data":"2bf6402bd8523e6501d1414f8050329b46c14e6014e98dce5b15adcd04074041"} Jul 06 19:30:00.197200 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:00.196862 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bf6402bd8523e6501d1414f8050329b46c14e6014e98dce5b15adcd04074041" Jul 06 19:30:00.198724 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:00.198700 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" Jul 06 19:30:00.198724 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:00.198706 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc" event={"ID":"0470f7d5-5f99-4019-9eea-336475adb661","Type":"ContainerDied","Data":"90c87dc2c7040f1e94eef4ba144c81307feced33267de32f89ffce870e2f3829"} Jul 06 19:30:00.198876 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:00.198737 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90c87dc2c7040f1e94eef4ba144c81307feced33267de32f89ffce870e2f3829" Jul 06 19:30:08.240138 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:08.240111 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:30:08.240798 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:08.240773 2573 generic.go:356] "Generic (PLEG): container finished" podID="7fbeec71-aed1-4625-9030-2431a05a7145" containerID="f8f81fd6d58322e826f9233a22182dd589b16e17900d1fa3eb53137e97bc7437" exitCode=1 Jul 06 19:30:08.240869 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:08.240844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" event={"ID":"7fbeec71-aed1-4625-9030-2431a05a7145","Type":"ContainerDied","Data":"f8f81fd6d58322e826f9233a22182dd589b16e17900d1fa3eb53137e97bc7437"} Jul 06 19:30:09.372832 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.372809 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:30:09.376109 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.376090 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:30:09.426352 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.426310 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-bundle\") pod \"7fbeec71-aed1-4625-9030-2431a05a7145\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " Jul 06 19:30:09.426352 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.426345 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgz5n\" (UniqueName: \"kubernetes.io/projected/7fbeec71-aed1-4625-9030-2431a05a7145-kube-api-access-sgz5n\") pod \"7fbeec71-aed1-4625-9030-2431a05a7145\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " Jul 06 19:30:09.426611 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.426385 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-util\") pod \"7fbeec71-aed1-4625-9030-2431a05a7145\" (UID: \"7fbeec71-aed1-4625-9030-2431a05a7145\") " Jul 06 19:30:09.427005 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.426974 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-bundle" (OuterVolumeSpecName: "bundle") pod "7fbeec71-aed1-4625-9030-2431a05a7145" (UID: "7fbeec71-aed1-4625-9030-2431a05a7145"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:30:09.428482 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.428457 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbeec71-aed1-4625-9030-2431a05a7145-kube-api-access-sgz5n" (OuterVolumeSpecName: "kube-api-access-sgz5n") pod "7fbeec71-aed1-4625-9030-2431a05a7145" (UID: "7fbeec71-aed1-4625-9030-2431a05a7145"). InnerVolumeSpecName "kube-api-access-sgz5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:30:09.431213 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.431188 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-util" (OuterVolumeSpecName: "util") pod "7fbeec71-aed1-4625-9030-2431a05a7145" (UID: "7fbeec71-aed1-4625-9030-2431a05a7145"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:30:09.527631 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.527538 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-util\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:30:09.527631 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.527567 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fbeec71-aed1-4625-9030-2431a05a7145-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:30:09.527631 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:09.527577 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgz5n\" (UniqueName: \"kubernetes.io/projected/7fbeec71-aed1-4625-9030-2431a05a7145-kube-api-access-sgz5n\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:30:10.251674 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:10.251628 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:30:10.252388 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:10.252366 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" event={"ID":"7fbeec71-aed1-4625-9030-2431a05a7145","Type":"ContainerDied","Data":"95bb1eb1e8400e3530ab716616454b376578100c7a09512266a8091645608def"} Jul 06 19:30:10.252459 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:10.252410 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95bb1eb1e8400e3530ab716616454b376578100c7a09512266a8091645608def" Jul 06 19:30:10.252459 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:10.252376 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns" Jul 06 19:30:17.062496 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062461 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq"] Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062763 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0470f7d5-5f99-4019-9eea-336475adb661" containerName="extract" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062774 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0470f7d5-5f99-4019-9eea-336475adb661" containerName="extract" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062783 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbeec71-aed1-4625-9030-2431a05a7145" containerName="extract" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062788 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbeec71-aed1-4625-9030-2431a05a7145" containerName="extract" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062794 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0470f7d5-5f99-4019-9eea-336475adb661" containerName="util" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062800 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0470f7d5-5f99-4019-9eea-336475adb661" containerName="util" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062804 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbeec71-aed1-4625-9030-2431a05a7145" containerName="pull" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062811 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbeec71-aed1-4625-9030-2431a05a7145" containerName="pull" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062821 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0470f7d5-5f99-4019-9eea-336475adb661" containerName="pull" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062827 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0470f7d5-5f99-4019-9eea-336475adb661" containerName="pull" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062833 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerName="extract" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062837 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerName="extract" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062844 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerName="util" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062849 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerName="util" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062855 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerName="extract" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062859 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerName="extract" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062867 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerName="pull" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062872 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerName="pull" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062879 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerName="pull" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062883 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerName="pull" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062889 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerName="util" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062894 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerName="util" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062898 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbeec71-aed1-4625-9030-2431a05a7145" containerName="util" Jul 06 19:30:17.062901 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062906 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbeec71-aed1-4625-9030-2431a05a7145" containerName="util" Jul 06 19:30:17.063704 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062953 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b05a45cd-a8d4-4e53-9a0e-d373374a70e0" containerName="extract" Jul 06 19:30:17.063704 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062963 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbeec71-aed1-4625-9030-2431a05a7145" containerName="extract" Jul 06 19:30:17.063704 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062970 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0470f7d5-5f99-4019-9eea-336475adb661" containerName="extract" Jul 06 19:30:17.063704 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.062975 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1523ea4a-cced-42ce-b7d2-532ec4baf1f5" containerName="extract" Jul 06 19:30:17.066365 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.066349 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.068952 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.068927 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:30:17.069135 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.069120 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mg4xx\"" Jul 06 19:30:17.069674 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.069656 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:30:17.086960 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.086930 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq"] Jul 06 19:30:17.097644 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.097617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.097745 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.097682 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvlhw\" (UniqueName: \"kubernetes.io/projected/bbaeecaa-91be-4793-8228-e685b108c266-kube-api-access-nvlhw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.097745 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.097715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.199120 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.199079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvlhw\" (UniqueName: \"kubernetes.io/projected/bbaeecaa-91be-4793-8228-e685b108c266-kube-api-access-nvlhw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.199120 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.199122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.199409 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.199250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.199484 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.199457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.199546 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.199528 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.209368 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.209345 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvlhw\" (UniqueName: \"kubernetes.io/projected/bbaeecaa-91be-4793-8228-e685b108c266-kube-api-access-nvlhw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.375348 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.375319 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:17.513313 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:17.513283 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq"] Jul 06 19:30:17.517127 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:30:17.517101 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbaeecaa_91be_4793_8228_e685b108c266.slice/crio-c26666886b8a92734f05c89087588bb94330ab510cdb0cf1e48110fb6c97b239 WatchSource:0}: Error finding container c26666886b8a92734f05c89087588bb94330ab510cdb0cf1e48110fb6c97b239: Status 404 returned error can't find the container with id c26666886b8a92734f05c89087588bb94330ab510cdb0cf1e48110fb6c97b239 Jul 06 19:30:18.289648 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:18.289617 2573 generic.go:356] "Generic (PLEG): container finished" podID="bbaeecaa-91be-4793-8228-e685b108c266" containerID="a1a41d91b70f2972c64a82bbe33dd6bd4f7fea4368af71318a32cb7d53662e11" exitCode=0 Jul 06 19:30:18.290062 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:18.289694 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" event={"ID":"bbaeecaa-91be-4793-8228-e685b108c266","Type":"ContainerDied","Data":"a1a41d91b70f2972c64a82bbe33dd6bd4f7fea4368af71318a32cb7d53662e11"} Jul 06 19:30:18.290062 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:18.289741 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" event={"ID":"bbaeecaa-91be-4793-8228-e685b108c266","Type":"ContainerStarted","Data":"c26666886b8a92734f05c89087588bb94330ab510cdb0cf1e48110fb6c97b239"} Jul 06 19:30:19.295794 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:19.295762 2573 generic.go:356] "Generic (PLEG): container finished" podID="bbaeecaa-91be-4793-8228-e685b108c266" containerID="ce2122621dd5f40c5876946ce867ceb9d965303d06e3b7daa6cb2f1623752e32" exitCode=0 Jul 06 19:30:19.296321 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:19.295819 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" event={"ID":"bbaeecaa-91be-4793-8228-e685b108c266","Type":"ContainerDied","Data":"ce2122621dd5f40c5876946ce867ceb9d965303d06e3b7daa6cb2f1623752e32"} Jul 06 19:30:20.304094 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:20.304054 2573 generic.go:356] "Generic (PLEG): container finished" podID="bbaeecaa-91be-4793-8228-e685b108c266" containerID="d6cd8f2358cf60faa7a58cfe0aa7dbb22cdd4639ae1f6778e79d32a51c954c41" exitCode=0 Jul 06 19:30:20.304527 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:20.304133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" event={"ID":"bbaeecaa-91be-4793-8228-e685b108c266","Type":"ContainerDied","Data":"d6cd8f2358cf60faa7a58cfe0aa7dbb22cdd4639ae1f6778e79d32a51c954c41"} Jul 06 19:30:21.439292 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.439269 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:21.535003 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.534957 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvlhw\" (UniqueName: \"kubernetes.io/projected/bbaeecaa-91be-4793-8228-e685b108c266-kube-api-access-nvlhw\") pod \"bbaeecaa-91be-4793-8228-e685b108c266\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " Jul 06 19:30:21.535176 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.535029 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-util\") pod \"bbaeecaa-91be-4793-8228-e685b108c266\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " Jul 06 19:30:21.535176 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.535137 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-bundle\") pod \"bbaeecaa-91be-4793-8228-e685b108c266\" (UID: \"bbaeecaa-91be-4793-8228-e685b108c266\") " Jul 06 19:30:21.535677 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.535654 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-bundle" (OuterVolumeSpecName: "bundle") pod "bbaeecaa-91be-4793-8228-e685b108c266" (UID: "bbaeecaa-91be-4793-8228-e685b108c266"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:30:21.537092 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.537070 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbaeecaa-91be-4793-8228-e685b108c266-kube-api-access-nvlhw" (OuterVolumeSpecName: "kube-api-access-nvlhw") pod "bbaeecaa-91be-4793-8228-e685b108c266" (UID: "bbaeecaa-91be-4793-8228-e685b108c266"). InnerVolumeSpecName "kube-api-access-nvlhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:30:21.540812 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.540775 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-util" (OuterVolumeSpecName: "util") pod "bbaeecaa-91be-4793-8228-e685b108c266" (UID: "bbaeecaa-91be-4793-8228-e685b108c266"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:30:21.635965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.635918 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:30:21.635965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.635961 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvlhw\" (UniqueName: \"kubernetes.io/projected/bbaeecaa-91be-4793-8228-e685b108c266-kube-api-access-nvlhw\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:30:21.635965 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:21.635973 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbaeecaa-91be-4793-8228-e685b108c266-util\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:30:22.316335 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:22.316298 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" Jul 06 19:30:22.316493 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:22.316298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq" event={"ID":"bbaeecaa-91be-4793-8228-e685b108c266","Type":"ContainerDied","Data":"c26666886b8a92734f05c89087588bb94330ab510cdb0cf1e48110fb6c97b239"} Jul 06 19:30:22.316493 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:22.316444 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26666886b8a92734f05c89087588bb94330ab510cdb0cf1e48110fb6c97b239" Jul 06 19:30:36.825815 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.825778 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56cf7cd8-vc72j"] Jul 06 19:30:36.826260 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.826100 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbaeecaa-91be-4793-8228-e685b108c266" containerName="util" Jul 06 19:30:36.826260 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.826111 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbaeecaa-91be-4793-8228-e685b108c266" containerName="util" Jul 06 19:30:36.826260 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.826128 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbaeecaa-91be-4793-8228-e685b108c266" containerName="extract" Jul 06 19:30:36.826260 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.826134 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbaeecaa-91be-4793-8228-e685b108c266" containerName="extract" Jul 06 19:30:36.826260 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.826141 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbaeecaa-91be-4793-8228-e685b108c266" containerName="pull" Jul 06 19:30:36.826260 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.826146 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbaeecaa-91be-4793-8228-e685b108c266" containerName="pull" Jul 06 19:30:36.826260 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.826200 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbaeecaa-91be-4793-8228-e685b108c266" containerName="extract" Jul 06 19:30:36.830596 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.830579 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:36.843295 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.843270 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56cf7cd8-vc72j"] Jul 06 19:30:36.970226 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.970164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-console-oauth-config\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:36.970226 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.970223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-console-config\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:36.970428 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.970278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-oauth-serving-cert\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:36.970428 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.970303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-trusted-ca-bundle\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:36.970428 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.970321 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-service-ca\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:36.970428 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.970353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-console-serving-cert\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:36.970428 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:36.970410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klps4\" (UniqueName: \"kubernetes.io/projected/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-kube-api-access-klps4\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.071150 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.071111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-oauth-serving-cert\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.071306 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.071166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-trusted-ca-bundle\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.071306 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.071190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-service-ca\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.071306 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.071265 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-console-serving-cert\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.071306 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.071297 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klps4\" (UniqueName: \"kubernetes.io/projected/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-kube-api-access-klps4\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.071454 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.071364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-console-oauth-config\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.071454 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.071386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-console-config\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.071997 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.071960 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-service-ca\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.072115 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.072023 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-oauth-serving-cert\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.072115 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.072027 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-console-config\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.072115 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.072090 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-trusted-ca-bundle\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.073826 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.073804 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-console-serving-cert\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.073902 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.073835 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-console-oauth-config\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.082332 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.082275 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klps4\" (UniqueName: \"kubernetes.io/projected/e0e3c94a-f1d6-4758-8ea1-e093d82a1459-kube-api-access-klps4\") pod \"console-56cf7cd8-vc72j\" (UID: \"e0e3c94a-f1d6-4758-8ea1-e093d82a1459\") " pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.139191 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.139151 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:37.282357 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.282334 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56cf7cd8-vc72j"] Jul 06 19:30:37.284524 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:30:37.284495 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e3c94a_f1d6_4758_8ea1_e093d82a1459.slice/crio-3b1716709fa49d917e3d6fde5d18c0218b7b9d8e6e0008f1338ad40b3d1f91f6 WatchSource:0}: Error finding container 3b1716709fa49d917e3d6fde5d18c0218b7b9d8e6e0008f1338ad40b3d1f91f6: Status 404 returned error can't find the container with id 3b1716709fa49d917e3d6fde5d18c0218b7b9d8e6e0008f1338ad40b3d1f91f6 Jul 06 19:30:37.389618 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.389583 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cf7cd8-vc72j" event={"ID":"e0e3c94a-f1d6-4758-8ea1-e093d82a1459","Type":"ContainerStarted","Data":"59281d620bf79aefee3ef7d988cb18929278b1ceba4030a0832667fa8292344a"} Jul 06 19:30:37.389806 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.389634 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cf7cd8-vc72j" event={"ID":"e0e3c94a-f1d6-4758-8ea1-e093d82a1459","Type":"ContainerStarted","Data":"3b1716709fa49d917e3d6fde5d18c0218b7b9d8e6e0008f1338ad40b3d1f91f6"} Jul 06 19:30:37.413716 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:37.413665 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56cf7cd8-vc72j" podStartSLOduration=1.413645887 podStartE2EDuration="1.413645887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:37.411840127 +0000 UTC m=+461.509067881" watchObservedRunningTime="2026-07-06 19:30:37.413645887 +0000 UTC m=+461.510873642" Jul 06 19:30:47.139539 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:47.139497 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:47.139918 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:47.139587 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:47.144392 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:47.144368 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:47.440559 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:47.440470 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56cf7cd8-vc72j" Jul 06 19:30:47.563340 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:47.563304 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66846d6df4-fh6f4"] Jul 06 19:30:56.841775 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:56.841740 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl"] Jul 06 19:30:56.845148 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:56.845130 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:30:56.847588 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:56.847565 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:30:56.847588 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:56.847567 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tdzvj\"" Jul 06 19:30:56.847959 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:56.847944 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:30:56.865370 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:56.865338 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl"] Jul 06 19:30:56.949784 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:56.949750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfhdv\" (UniqueName: \"kubernetes.io/projected/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-kube-api-access-wfhdv\") pod \"kuadrant-operator-controller-manager-86878c769f-tvsgl\" (UID: \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:30:56.949784 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:56.949788 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tvsgl\" (UID: \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:30:57.050338 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:57.050294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfhdv\" (UniqueName: \"kubernetes.io/projected/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-kube-api-access-wfhdv\") pod \"kuadrant-operator-controller-manager-86878c769f-tvsgl\" (UID: \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:30:57.050338 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:57.050337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tvsgl\" (UID: \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:30:57.050713 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:57.050695 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tvsgl\" (UID: \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:30:57.064536 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:57.064502 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfhdv\" (UniqueName: \"kubernetes.io/projected/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-kube-api-access-wfhdv\") pod \"kuadrant-operator-controller-manager-86878c769f-tvsgl\" (UID: \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:30:57.155958 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:57.155864 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:30:57.319988 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:57.319959 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl"] Jul 06 19:30:57.322174 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:30:57.322134 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef47e9b2_f7de_4c96_b7f4_0c83f689b9f6.slice/crio-ec0330d7caa9bc5a0ef6e1da89c5fb2392188c347c4a813712e514da534ea4f3 WatchSource:0}: Error finding container ec0330d7caa9bc5a0ef6e1da89c5fb2392188c347c4a813712e514da534ea4f3: Status 404 returned error can't find the container with id ec0330d7caa9bc5a0ef6e1da89c5fb2392188c347c4a813712e514da534ea4f3 Jul 06 19:30:57.486014 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:30:57.485932 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerStarted","Data":"ec0330d7caa9bc5a0ef6e1da89c5fb2392188c347c4a813712e514da534ea4f3"} Jul 06 19:31:03.517605 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:03.517568 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerStarted","Data":"aaff948e6d96a40e506b4079e6c6644863d8c4a5b463a5ca7e4ff7b486c8c551"} Jul 06 19:31:03.518002 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:03.517696 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:31:12.583979 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.583896 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-66846d6df4-fh6f4" podUID="9bde0ff7-a45f-4cc2-8372-01d0e271a46c" containerName="console" containerID="cri-o://c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267" gracePeriod=15 Jul 06 19:31:12.824426 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.824395 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:31:12.859050 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.858957 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" podStartSLOduration=11.680107686 podStartE2EDuration="16.85894094s" totalImagesPullingTime="5.178833254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:31:03.580718343 +0000 UTC m=+487.677946098" watchObservedRunningTime="2026-07-06 19:31:12.85894094 +0000 UTC m=+496.956168695" Jul 06 19:31:12.897764 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.897734 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-oauth-config\") pod \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " Jul 06 19:31:12.897970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.897777 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-serving-cert\") pod \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " Jul 06 19:31:12.897970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.897799 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6475\" (UniqueName: \"kubernetes.io/projected/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-kube-api-access-s6475\") pod \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " Jul 06 19:31:12.897970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.897842 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-trusted-ca-bundle\") pod \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " Jul 06 19:31:12.897970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.897899 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-service-ca\") pod \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " Jul 06 19:31:12.897970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.897958 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-oauth-serving-cert\") pod \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " Jul 06 19:31:12.898259 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.897985 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-config\") pod \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\" (UID: \"9bde0ff7-a45f-4cc2-8372-01d0e271a46c\") " Jul 06 19:31:12.898458 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.898404 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-service-ca" (OuterVolumeSpecName: "service-ca") pod "9bde0ff7-a45f-4cc2-8372-01d0e271a46c" (UID: "9bde0ff7-a45f-4cc2-8372-01d0e271a46c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:31:12.898458 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.898423 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9bde0ff7-a45f-4cc2-8372-01d0e271a46c" (UID: "9bde0ff7-a45f-4cc2-8372-01d0e271a46c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:31:12.898637 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.898555 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9bde0ff7-a45f-4cc2-8372-01d0e271a46c" (UID: "9bde0ff7-a45f-4cc2-8372-01d0e271a46c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:31:12.898637 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.898590 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-config" (OuterVolumeSpecName: "console-config") pod "9bde0ff7-a45f-4cc2-8372-01d0e271a46c" (UID: "9bde0ff7-a45f-4cc2-8372-01d0e271a46c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:31:12.900038 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.900011 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9bde0ff7-a45f-4cc2-8372-01d0e271a46c" (UID: "9bde0ff7-a45f-4cc2-8372-01d0e271a46c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:31:12.900151 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.900107 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9bde0ff7-a45f-4cc2-8372-01d0e271a46c" (UID: "9bde0ff7-a45f-4cc2-8372-01d0e271a46c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:31:12.900151 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.900129 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-kube-api-access-s6475" (OuterVolumeSpecName: "kube-api-access-s6475") pod "9bde0ff7-a45f-4cc2-8372-01d0e271a46c" (UID: "9bde0ff7-a45f-4cc2-8372-01d0e271a46c"). InnerVolumeSpecName "kube-api-access-s6475". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:31:12.999373 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.999336 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-oauth-serving-cert\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:31:12.999373 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.999366 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:31:12.999373 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.999375 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-oauth-config\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:31:12.999373 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.999383 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-console-serving-cert\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:31:12.999636 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.999392 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6475\" (UniqueName: \"kubernetes.io/projected/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-kube-api-access-s6475\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:31:12.999636 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.999401 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-trusted-ca-bundle\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:31:12.999636 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:12.999409 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bde0ff7-a45f-4cc2-8372-01d0e271a46c-service-ca\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:31:13.573795 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:13.573758 2573 generic.go:356] "Generic (PLEG): container finished" podID="9bde0ff7-a45f-4cc2-8372-01d0e271a46c" containerID="c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267" exitCode=0 Jul 06 19:31:13.574007 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:13.573804 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66846d6df4-fh6f4" event={"ID":"9bde0ff7-a45f-4cc2-8372-01d0e271a46c","Type":"ContainerDied","Data":"c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267"} Jul 06 19:31:13.574007 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:13.573809 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66846d6df4-fh6f4" Jul 06 19:31:13.574007 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:13.573844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66846d6df4-fh6f4" event={"ID":"9bde0ff7-a45f-4cc2-8372-01d0e271a46c","Type":"ContainerDied","Data":"c1648ce72098ec389f1ccd654326706cff6f526b8134f8f60b154aca3405b350"} Jul 06 19:31:13.574007 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:13.573863 2573 scope.go:117] "RemoveContainer" containerID="c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267" Jul 06 19:31:13.583608 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:13.583591 2573 scope.go:117] "RemoveContainer" containerID="c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267" Jul 06 19:31:13.583882 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:31:13.583860 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267\": container with ID starting with c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267 not found: ID does not exist" containerID="c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267" Jul 06 19:31:13.583970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:13.583887 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267"} err="failed to get container status \"c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267\": rpc error: code = NotFound desc = could not find container \"c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267\": container with ID starting with c7c52a3d53a669569df54ccf8457eb2902f474413872b01b3a85fdac24a42267 not found: ID does not exist" Jul 06 19:31:13.612242 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:13.612188 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66846d6df4-fh6f4"] Jul 06 19:31:13.628025 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:13.627999 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66846d6df4-fh6f4"] Jul 06 19:31:14.394151 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:14.394119 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bde0ff7-a45f-4cc2-8372-01d0e271a46c" path="/var/lib/kubelet/pods/9bde0ff7-a45f-4cc2-8372-01d0e271a46c/volumes" Jul 06 19:31:14.525663 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:31:14.525632 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:32:56.281795 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:56.281768 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:32:56.286165 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:56.286146 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:32:57.695560 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.695528 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-g25zj"] Jul 06 19:32:57.695970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.695863 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bde0ff7-a45f-4cc2-8372-01d0e271a46c" containerName="console" Jul 06 19:32:57.695970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.695874 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bde0ff7-a45f-4cc2-8372-01d0e271a46c" containerName="console" Jul 06 19:32:57.695970 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.695947 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bde0ff7-a45f-4cc2-8372-01d0e271a46c" containerName="console" Jul 06 19:32:57.698448 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.698426 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:32:57.702246 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.702218 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pgc25\"" Jul 06 19:32:57.702414 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.702395 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 19:32:57.717337 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.717306 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-g25zj"] Jul 06 19:32:57.782413 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.782371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f046b51b-34bb-4e1b-aba0-88431a3f1422-plugins-config-volume\") pod \"payload-pre-processing-7964457696-g25zj\" (UID: \"f046b51b-34bb-4e1b-aba0-88431a3f1422\") " pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:32:57.782608 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.782434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9wkh\" (UniqueName: \"kubernetes.io/projected/f046b51b-34bb-4e1b-aba0-88431a3f1422-kube-api-access-x9wkh\") pod \"payload-pre-processing-7964457696-g25zj\" (UID: \"f046b51b-34bb-4e1b-aba0-88431a3f1422\") " pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:32:57.883657 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.883621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f046b51b-34bb-4e1b-aba0-88431a3f1422-plugins-config-volume\") pod \"payload-pre-processing-7964457696-g25zj\" (UID: \"f046b51b-34bb-4e1b-aba0-88431a3f1422\") " pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:32:57.883829 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.883667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9wkh\" (UniqueName: \"kubernetes.io/projected/f046b51b-34bb-4e1b-aba0-88431a3f1422-kube-api-access-x9wkh\") pod \"payload-pre-processing-7964457696-g25zj\" (UID: \"f046b51b-34bb-4e1b-aba0-88431a3f1422\") " pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:32:57.884370 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.884351 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f046b51b-34bb-4e1b-aba0-88431a3f1422-plugins-config-volume\") pod \"payload-pre-processing-7964457696-g25zj\" (UID: \"f046b51b-34bb-4e1b-aba0-88431a3f1422\") " pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:32:57.893976 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:57.893937 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9wkh\" (UniqueName: \"kubernetes.io/projected/f046b51b-34bb-4e1b-aba0-88431a3f1422-kube-api-access-x9wkh\") pod \"payload-pre-processing-7964457696-g25zj\" (UID: \"f046b51b-34bb-4e1b-aba0-88431a3f1422\") " pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:32:58.009042 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:58.008949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:32:58.149959 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:58.149929 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-g25zj"] Jul 06 19:32:58.150629 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:32:58.150600 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf046b51b_34bb_4e1b_aba0_88431a3f1422.slice/crio-b82e20f7c6776fb5dd3fdc96236239beebc7026eebebc6b6e687f95d756feecc WatchSource:0}: Error finding container b82e20f7c6776fb5dd3fdc96236239beebc7026eebebc6b6e687f95d756feecc: Status 404 returned error can't find the container with id b82e20f7c6776fb5dd3fdc96236239beebc7026eebebc6b6e687f95d756feecc Jul 06 19:32:59.102876 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:32:59.102842 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" event={"ID":"f046b51b-34bb-4e1b-aba0-88431a3f1422","Type":"ContainerStarted","Data":"b82e20f7c6776fb5dd3fdc96236239beebc7026eebebc6b6e687f95d756feecc"} Jul 06 19:33:00.109633 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:00.109596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" event={"ID":"f046b51b-34bb-4e1b-aba0-88431a3f1422","Type":"ContainerStarted","Data":"d722b36ad5559f74877d49bbcd9d35a9a4ea706275505992d793237ccf91267b"} Jul 06 19:33:00.110054 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:00.109655 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:33:00.132019 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:00.131966 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" podStartSLOduration=1.7290303759999999 podStartE2EDuration="3.131951167s" totalImagesPullingTime="1.402920791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:00.130238131 +0000 UTC m=+604.227465885" watchObservedRunningTime="2026-07-06 19:33:00.131951167 +0000 UTC m=+604.229178920" Jul 06 19:33:11.117082 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:11.117050 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-g25zj" Jul 06 19:33:22.614466 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.614393 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6"] Jul 06 19:33:22.617675 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.617658 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.620341 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.620319 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 19:33:22.620447 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.620349 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:33:22.620493 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.620447 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:33:22.620821 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.620803 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-spm8h\"" Jul 06 19:33:22.632074 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.632051 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6"] Jul 06 19:33:22.795073 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.795032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.795073 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.795077 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.795323 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.795174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnh2t\" (UniqueName: \"kubernetes.io/projected/5c1717cd-5944-4288-a859-6dca347a45b8-kube-api-access-pnh2t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.795323 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.795194 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1717cd-5944-4288-a859-6dca347a45b8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.795323 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.795268 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.795323 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.795285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.895910 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.895814 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnh2t\" (UniqueName: \"kubernetes.io/projected/5c1717cd-5944-4288-a859-6dca347a45b8-kube-api-access-pnh2t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.895910 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.895866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1717cd-5944-4288-a859-6dca347a45b8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.895910 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.895909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.896188 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.895926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.896188 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.895941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.896188 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.895962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.896386 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.896338 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.896441 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.896398 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.896441 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.896407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.898380 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.898356 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c1717cd-5944-4288-a859-6dca347a45b8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.898557 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.898536 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1717cd-5944-4288-a859-6dca347a45b8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.907960 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.907930 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnh2t\" (UniqueName: \"kubernetes.io/projected/5c1717cd-5944-4288-a859-6dca347a45b8-kube-api-access-pnh2t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6\" (UID: \"5c1717cd-5944-4288-a859-6dca347a45b8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:22.928331 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:22.928302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:23.066703 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:23.066633 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6"] Jul 06 19:33:23.068971 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:33:23.068946 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1717cd_5944_4288_a859_6dca347a45b8.slice/crio-eec3d02c420abb97cdd4d7a5e43fdcc0029a353e392d28f770459925905c090f WatchSource:0}: Error finding container eec3d02c420abb97cdd4d7a5e43fdcc0029a353e392d28f770459925905c090f: Status 404 returned error can't find the container with id eec3d02c420abb97cdd4d7a5e43fdcc0029a353e392d28f770459925905c090f Jul 06 19:33:23.070397 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:23.070377 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:33:23.223728 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:23.223632 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" event={"ID":"5c1717cd-5944-4288-a859-6dca347a45b8","Type":"ContainerStarted","Data":"eec3d02c420abb97cdd4d7a5e43fdcc0029a353e392d28f770459925905c090f"} Jul 06 19:33:24.237795 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:24.237756 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" event={"ID":"5c1717cd-5944-4288-a859-6dca347a45b8","Type":"ContainerStarted","Data":"7f605159d020fd402e2a5e0a7fa4600a236aff99a21f094abdc11ead482cc19a"} Jul 06 19:33:24.262185 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:24.262130 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" podStartSLOduration=1.248617388 podStartE2EDuration="2.262114772s" totalImagesPullingTime="1.013497384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:24.259116013 +0000 UTC m=+628.356343795" watchObservedRunningTime="2026-07-06 19:33:24.262114772 +0000 UTC m=+628.359342526" Jul 06 19:33:32.928777 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:32.928732 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:32.929187 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:32.928802 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:32.941564 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:32.941534 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:33:33.293929 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:33:33.293836 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6" Jul 06 19:34:09.819650 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:09.819613 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px"] Jul 06 19:34:09.823086 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:09.823066 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:09.827390 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:09.827374 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 19:34:09.839031 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:09.839008 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px"] Jul 06 19:34:10.006235 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.006187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.006433 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.006247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5rmz\" (UniqueName: \"kubernetes.io/projected/39f605cb-86b3-4cf2-be88-a7ef37924ec7-kube-api-access-b5rmz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.006433 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.006337 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f605cb-86b3-4cf2-be88-a7ef37924ec7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.006433 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.006371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.006534 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.006434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.006534 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.006460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.107284 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.107252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.107447 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.107297 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5rmz\" (UniqueName: \"kubernetes.io/projected/39f605cb-86b3-4cf2-be88-a7ef37924ec7-kube-api-access-b5rmz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.107447 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.107342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f605cb-86b3-4cf2-be88-a7ef37924ec7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.107447 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.107368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.107447 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.107413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.107447 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.107437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.107730 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.107708 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.107789 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.107752 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.107845 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.107793 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.109695 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.109672 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f605cb-86b3-4cf2-be88-a7ef37924ec7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.109977 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.109957 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f605cb-86b3-4cf2-be88-a7ef37924ec7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.116618 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.116586 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5rmz\" (UniqueName: \"kubernetes.io/projected/39f605cb-86b3-4cf2-be88-a7ef37924ec7-kube-api-access-b5rmz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v5px\" (UID: \"39f605cb-86b3-4cf2-be88-a7ef37924ec7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.133398 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.133364 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:10.281328 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.281290 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px"] Jul 06 19:34:10.282056 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:34:10.282027 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f605cb_86b3_4cf2_be88_a7ef37924ec7.slice/crio-9ec24422b3ac1c3787d57b29aafc9b3703b25b25d3c20709f2b60ce468f36a2e WatchSource:0}: Error finding container 9ec24422b3ac1c3787d57b29aafc9b3703b25b25d3c20709f2b60ce468f36a2e: Status 404 returned error can't find the container with id 9ec24422b3ac1c3787d57b29aafc9b3703b25b25d3c20709f2b60ce468f36a2e Jul 06 19:34:10.471586 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:10.471495 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" event={"ID":"39f605cb-86b3-4cf2-be88-a7ef37924ec7","Type":"ContainerStarted","Data":"9ec24422b3ac1c3787d57b29aafc9b3703b25b25d3c20709f2b60ce468f36a2e"} Jul 06 19:34:11.478030 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:11.477996 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" event={"ID":"39f605cb-86b3-4cf2-be88-a7ef37924ec7","Type":"ContainerStarted","Data":"0ef1506768778674c9241a7766facd4b9861a2c05df4acf713ab4bdc53983b6d"} Jul 06 19:34:11.511487 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:11.511438 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" podStartSLOduration=2.196849269 podStartE2EDuration="2.51142369s" totalImagesPullingTime="314.574421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:11.508093956 +0000 UTC m=+675.605321709" watchObservedRunningTime="2026-07-06 19:34:11.51142369 +0000 UTC m=+675.608651443" Jul 06 19:34:20.133929 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:20.133881 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:20.134469 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:20.133964 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:20.146585 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:20.146559 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:34:20.534728 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:34:20.534643 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v5px" Jul 06 19:35:53.106164 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:35:53.106121 2573 controller.go:195] "Failed to update lease" err="Put \"https://a7094b167326a4c82a83b6b901f5856c-474eb07a5e9b8c2e.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-140-103.ec2.internal?timeout=10s\": context deadline exceeded" Jul 06 19:35:54.006390 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:54.006361 2573 generic.go:356] "Generic (PLEG): container finished" podID="e17f75ab-ddd9-4fe5-8a4b-b8f969a45874" containerID="2fc019726d121432255fb0eab1b36d800623dee08afce4eddb2ab4d1dee1442e" exitCode=1 Jul 06 19:35:54.006504 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:54.006432 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" event={"ID":"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874","Type":"ContainerDied","Data":"2fc019726d121432255fb0eab1b36d800623dee08afce4eddb2ab4d1dee1442e"} Jul 06 19:35:54.006847 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:54.006827 2573 scope.go:117] "RemoveContainer" containerID="2fc019726d121432255fb0eab1b36d800623dee08afce4eddb2ab4d1dee1442e" Jul 06 19:35:54.008240 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:54.008141 2573 generic.go:356] "Generic (PLEG): container finished" podID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerID="aaff948e6d96a40e506b4079e6c6644863d8c4a5b463a5ca7e4ff7b486c8c551" exitCode=0 Jul 06 19:35:54.008240 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:54.008178 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerDied","Data":"aaff948e6d96a40e506b4079e6c6644863d8c4a5b463a5ca7e4ff7b486c8c551"} Jul 06 19:35:54.008477 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:54.008464 2573 scope.go:117] "RemoveContainer" containerID="aaff948e6d96a40e506b4079e6c6644863d8c4a5b463a5ca7e4ff7b486c8c551" Jul 06 19:35:54.524166 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:54.524126 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:35:55.015201 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:55.015161 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerStarted","Data":"964496c6ec28acecda60f97ea52c8c25d14fe3ca73ec51f5d4f1c032c8a0b8d7"} Jul 06 19:35:55.015418 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:55.015239 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:35:55.016876 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:55.016857 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" event={"ID":"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874","Type":"ContainerStarted","Data":"318bf7b32f4483b3c2f37d69f382be5e57ebf31a912666a1fc63fb911159de02"} Jul 06 19:35:55.017039 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:35:55.017024 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:35:57.966106 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:35:57.966068 2573 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-140-103.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again" Jul 06 19:36:06.024571 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:36:06.024543 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:36:06.024988 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:36:06.024609 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:37:56.317597 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:37:56.317507 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:37:56.318823 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:37:56.318800 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:40:40.486160 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:40.486066 2573 generic.go:356] "Generic (PLEG): container finished" podID="e17f75ab-ddd9-4fe5-8a4b-b8f969a45874" containerID="318bf7b32f4483b3c2f37d69f382be5e57ebf31a912666a1fc63fb911159de02" exitCode=1 Jul 06 19:40:40.486160 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:40.486141 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" event={"ID":"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874","Type":"ContainerDied","Data":"318bf7b32f4483b3c2f37d69f382be5e57ebf31a912666a1fc63fb911159de02"} Jul 06 19:40:40.486707 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:40.486228 2573 scope.go:117] "RemoveContainer" containerID="2fc019726d121432255fb0eab1b36d800623dee08afce4eddb2ab4d1dee1442e" Jul 06 19:40:40.486707 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:40.486544 2573 scope.go:117] "RemoveContainer" containerID="318bf7b32f4483b3c2f37d69f382be5e57ebf31a912666a1fc63fb911159de02" Jul 06 19:40:40.486800 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:40:40.486779 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=lws-controller-manager-54d6cd8878-9x5fq_openshift-lws-operator(e17f75ab-ddd9-4fe5-8a4b-b8f969a45874)\"" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" podUID="e17f75ab-ddd9-4fe5-8a4b-b8f969a45874" Jul 06 19:40:40.488328 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:40.488310 2573 generic.go:356] "Generic (PLEG): container finished" podID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerID="964496c6ec28acecda60f97ea52c8c25d14fe3ca73ec51f5d4f1c032c8a0b8d7" exitCode=0 Jul 06 19:40:40.488420 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:40.488347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerDied","Data":"964496c6ec28acecda60f97ea52c8c25d14fe3ca73ec51f5d4f1c032c8a0b8d7"} Jul 06 19:40:40.488630 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:40.488618 2573 scope.go:117] "RemoveContainer" containerID="964496c6ec28acecda60f97ea52c8c25d14fe3ca73ec51f5d4f1c032c8a0b8d7" Jul 06 19:40:40.488815 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:40:40.488799 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-tvsgl_kuadrant-system(ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" Jul 06 19:40:40.497290 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:40.497261 2573 scope.go:117] "RemoveContainer" containerID="aaff948e6d96a40e506b4079e6c6644863d8c4a5b463a5ca7e4ff7b486c8c551" Jul 06 19:40:46.022669 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:46.022627 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:40:46.023088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:46.022702 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:40:46.023088 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:46.023020 2573 scope.go:117] "RemoveContainer" containerID="964496c6ec28acecda60f97ea52c8c25d14fe3ca73ec51f5d4f1c032c8a0b8d7" Jul 06 19:40:46.023156 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:46.023105 2573 scope.go:117] "RemoveContainer" containerID="318bf7b32f4483b3c2f37d69f382be5e57ebf31a912666a1fc63fb911159de02" Jul 06 19:40:46.023250 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:40:46.023199 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-tvsgl_kuadrant-system(ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" Jul 06 19:40:46.023304 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:40:46.023256 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=lws-controller-manager-54d6cd8878-9x5fq_openshift-lws-operator(e17f75ab-ddd9-4fe5-8a4b-b8f969a45874)\"" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" podUID="e17f75ab-ddd9-4fe5-8a4b-b8f969a45874" Jul 06 19:40:56.355151 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:40:56.355045 2573 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-06T19:40:40Z\\\",\\\"message\\\":\\\"containers with unready status: [manager]\\\",\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-06T19:40:40Z\\\",\\\"message\\\":\\\"containers with unready status: [manager]\\\",\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"1\\\",\\\"memory\\\":\\\"1Gi\\\"},\\\"containerID\\\":\\\"cri-o://318bf7b32f4483b3c2f37d69f382be5e57ebf31a912666a1fc63fb911159de02\\\",\\\"image\\\":\\\"registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f\\\",\\\"imageID\\\":\\\"registry.redhat.io/leader-worker-set/lws-rhel9@sha256:70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://2fc019726d121432255fb0eab1b36d800623dee08afce4eddb2ab4d1dee1442e\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-07-06T19:35:53Z\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-06T19:28:42Z\\\"}},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"1\\\",\\\"memory\\\":\\\"1Gi\\\"}},\\\"restartCount\\\":1,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://318bf7b32f4483b3c2f37d69f382be5e57ebf31a912666a1fc63fb911159de02\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-07-06T19:40:40Z\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-06T19:35:54Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000640000],\\\"uid\\\":1000640000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/controller_manager_config.yaml\\\",\\\"name\\\":\\\"manager-config\\\"},{\\\"mountPath\\\":\\\"/tmp/k8s-webhook-server/serving-certs\\\",\\\"name\\\":\\\"cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/tmp/k8s-metrics-server/serving-certs\\\",\\\"name\\\":\\\"metrics-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-8cnwq\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-lws-operator\"/\"lws-controller-manager-54d6cd8878-9x5fq\": Internal error occurred: admission plugin \"ValidatingAdmissionWebhook\" failed to complete validation in 13s" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:40:57.156172 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:57.156140 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:40:57.156527 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:57.156515 2573 scope.go:117] "RemoveContainer" containerID="964496c6ec28acecda60f97ea52c8c25d14fe3ca73ec51f5d4f1c032c8a0b8d7" Jul 06 19:40:57.585104 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:57.585072 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerStarted","Data":"635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6"} Jul 06 19:40:57.585522 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:57.585322 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:40:59.673916 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:59.673863 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:40:59.674355 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:40:59.674303 2573 scope.go:117] "RemoveContainer" containerID="318bf7b32f4483b3c2f37d69f382be5e57ebf31a912666a1fc63fb911159de02" Jul 06 19:41:00.600796 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:00.600759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" event={"ID":"e17f75ab-ddd9-4fe5-8a4b-b8f969a45874","Type":"ContainerStarted","Data":"b1cb7eabf1fced2bfde1bdfb505b53e1cbaf1815d1587c0b5ef619ce570eed1e"} Jul 06 19:41:00.600984 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:00.600970 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:41:08.591624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:08.591593 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:41:11.607952 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:11.607922 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54d6cd8878-9x5fq" Jul 06 19:41:33.771587 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:33.771555 2573 generic.go:356] "Generic (PLEG): container finished" podID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerID="635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6" exitCode=0 Jul 06 19:41:33.772044 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:33.771605 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerDied","Data":"635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6"} Jul 06 19:41:33.772044 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:33.771658 2573 scope.go:117] "RemoveContainer" containerID="964496c6ec28acecda60f97ea52c8c25d14fe3ca73ec51f5d4f1c032c8a0b8d7" Jul 06 19:41:33.772044 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:33.771976 2573 scope.go:117] "RemoveContainer" containerID="635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6" Jul 06 19:41:33.772192 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:41:33.772172 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-tvsgl_kuadrant-system(ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" Jul 06 19:41:34.248225 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:41:34.248164 2573 controller.go:195] "Failed to update lease" err="Put \"https://a7094b167326a4c82a83b6b901f5856c-474eb07a5e9b8c2e.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-140-103.ec2.internal?timeout=10s\": context deadline exceeded" Jul 06 19:41:37.156863 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:37.156832 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:41:37.157287 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:37.157200 2573 scope.go:117] "RemoveContainer" containerID="635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6" Jul 06 19:41:37.157405 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:41:37.157387 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-tvsgl_kuadrant-system(ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" Jul 06 19:41:38.590755 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:38.590714 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:41:38.591150 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:38.591099 2573 scope.go:117] "RemoveContainer" containerID="635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6" Jul 06 19:41:38.591315 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:41:38.591295 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-tvsgl_kuadrant-system(ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" Jul 06 19:41:50.389545 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:41:50.389509 2573 scope.go:117] "RemoveContainer" containerID="635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6" Jul 06 19:41:50.390034 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:41:50.389756 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-tvsgl_kuadrant-system(ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" Jul 06 19:42:01.388668 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:42:01.388638 2573 scope.go:117] "RemoveContainer" containerID="635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6" Jul 06 19:42:01.914013 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:42:01.913975 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerStarted","Data":"3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b"} Jul 06 19:42:01.914201 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:42:01.914183 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:42:12.921335 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:42:12.921305 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:42:56.350699 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:42:56.350665 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:42:56.351245 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:42:56.350715 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:46:07.787588 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:07.787551 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl"] Jul 06 19:46:07.790604 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:07.787762 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" containerID="cri-o://3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b" gracePeriod=10 Jul 06 19:46:10.448878 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:10.448855 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:46:10.566985 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:10.566914 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-extensions-socket-volume\") pod \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\" (UID: \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\") " Jul 06 19:46:10.566985 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:10.566953 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfhdv\" (UniqueName: \"kubernetes.io/projected/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-kube-api-access-wfhdv\") pod \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\" (UID: \"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6\") " Jul 06 19:46:10.567355 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:10.567332 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" (UID: "ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:10.568935 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:10.568916 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-kube-api-access-wfhdv" (OuterVolumeSpecName: "kube-api-access-wfhdv") pod "ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" (UID: "ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6"). InnerVolumeSpecName "kube-api-access-wfhdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:46:10.667726 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:10.667691 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-extensions-socket-volume\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:46:10.667726 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:10.667719 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfhdv\" (UniqueName: \"kubernetes.io/projected/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6-kube-api-access-wfhdv\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:46:11.162536 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.162501 2573 generic.go:356] "Generic (PLEG): container finished" podID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerID="3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b" exitCode=0 Jul 06 19:46:11.162701 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.162565 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" Jul 06 19:46:11.162701 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.162586 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerDied","Data":"3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b"} Jul 06 19:46:11.162701 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.162675 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl" event={"ID":"ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6","Type":"ContainerDied","Data":"ec0330d7caa9bc5a0ef6e1da89c5fb2392188c347c4a813712e514da534ea4f3"} Jul 06 19:46:11.162804 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.162704 2573 scope.go:117] "RemoveContainer" containerID="3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b" Jul 06 19:46:11.175785 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.175767 2573 scope.go:117] "RemoveContainer" containerID="635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6" Jul 06 19:46:11.183995 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.183975 2573 scope.go:117] "RemoveContainer" containerID="3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b" Jul 06 19:46:11.184266 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:46:11.184247 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b\": container with ID starting with 3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b not found: ID does not exist" containerID="3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b" Jul 06 19:46:11.184344 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.184285 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b"} err="failed to get container status \"3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b\": rpc error: code = NotFound desc = could not find container \"3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b\": container with ID starting with 3595e34ef6284543df040178487b3467d4a0ff64e80fc2f5062f6f1085bbc08b not found: ID does not exist" Jul 06 19:46:11.184344 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.184310 2573 scope.go:117] "RemoveContainer" containerID="635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6" Jul 06 19:46:11.184551 ip-10-0-140-103 kubenswrapper[2573]: E0706 19:46:11.184534 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6\": container with ID starting with 635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6 not found: ID does not exist" containerID="635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6" Jul 06 19:46:11.184588 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.184557 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6"} err="failed to get container status \"635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6\": rpc error: code = NotFound desc = could not find container \"635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6\": container with ID starting with 635319c391d3308f0fa4e688b18c042ee6c0c84b31439f713e9a6b07839772c6 not found: ID does not exist" Jul 06 19:46:11.200660 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.200633 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl"] Jul 06 19:46:11.204598 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:11.204573 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tvsgl"] Jul 06 19:46:12.394347 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:46:12.394315 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" path="/var/lib/kubelet/pods/ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6/volumes" Jul 06 19:47:08.881001 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.880966 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2"] Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881328 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881343 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881357 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881362 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881369 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881373 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881435 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881443 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881451 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.881450 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.881458 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:47:08.885581 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.885564 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:08.892688 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.892669 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:47:08.892774 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.892726 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tdzvj\"" Jul 06 19:47:08.892948 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.892933 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:47:08.919679 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.919647 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2"] Jul 06 19:47:08.953005 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.952970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94247084-7c3d-4b76-a4b9-72d708348353-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x97z2\" (UID: \"94247084-7c3d-4b76-a4b9-72d708348353\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:08.953175 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:08.953064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqh7\" (UniqueName: \"kubernetes.io/projected/94247084-7c3d-4b76-a4b9-72d708348353-kube-api-access-5wqh7\") pod \"kuadrant-operator-controller-manager-86878c769f-x97z2\" (UID: \"94247084-7c3d-4b76-a4b9-72d708348353\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:09.053807 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.053768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94247084-7c3d-4b76-a4b9-72d708348353-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x97z2\" (UID: \"94247084-7c3d-4b76-a4b9-72d708348353\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:09.053807 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.053821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqh7\" (UniqueName: \"kubernetes.io/projected/94247084-7c3d-4b76-a4b9-72d708348353-kube-api-access-5wqh7\") pod \"kuadrant-operator-controller-manager-86878c769f-x97z2\" (UID: \"94247084-7c3d-4b76-a4b9-72d708348353\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:09.054175 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.054154 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94247084-7c3d-4b76-a4b9-72d708348353-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x97z2\" (UID: \"94247084-7c3d-4b76-a4b9-72d708348353\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:09.065973 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.065948 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqh7\" (UniqueName: \"kubernetes.io/projected/94247084-7c3d-4b76-a4b9-72d708348353-kube-api-access-5wqh7\") pod \"kuadrant-operator-controller-manager-86878c769f-x97z2\" (UID: \"94247084-7c3d-4b76-a4b9-72d708348353\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:09.196281 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.196184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:09.329461 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.329437 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2"] Jul 06 19:47:09.331640 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:47:09.331613 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94247084_7c3d_4b76_a4b9_72d708348353.slice/crio-8d1c095673d64d5ee6330df437b44f495d99101ee64c91e27b7c3fbcb2d08013 WatchSource:0}: Error finding container 8d1c095673d64d5ee6330df437b44f495d99101ee64c91e27b7c3fbcb2d08013: Status 404 returned error can't find the container with id 8d1c095673d64d5ee6330df437b44f495d99101ee64c91e27b7c3fbcb2d08013 Jul 06 19:47:09.458700 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.458632 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" event={"ID":"94247084-7c3d-4b76-a4b9-72d708348353","Type":"ContainerStarted","Data":"3361e0bddfc4cd2cb59f7a2c457dfb0397da1d4cbead6e77b09ab4e38d9764ef"} Jul 06 19:47:09.458700 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.458678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" event={"ID":"94247084-7c3d-4b76-a4b9-72d708348353","Type":"ContainerStarted","Data":"8d1c095673d64d5ee6330df437b44f495d99101ee64c91e27b7c3fbcb2d08013"} Jul 06 19:47:09.458873 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.458761 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:09.485882 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:09.485836 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" podStartSLOduration=1.485823954 podStartE2EDuration="1.485823954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:47:09.484086202 +0000 UTC m=+1453.581313969" watchObservedRunningTime="2026-07-06 19:47:09.485823954 +0000 UTC m=+1453.583051708" Jul 06 19:47:20.465712 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:20.465683 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x97z2" Jul 06 19:47:56.382281 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:56.382251 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:47:56.383549 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:47:56.383529 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:52:56.415304 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:52:56.415278 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:52:56.417106 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:52:56.417080 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:56:58.735156 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.735124 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8"] Jul 06 19:56:58.735752 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.735466 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:56:58.735752 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.735478 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef47e9b2-f7de-4c96-b7f4-0c83f689b9f6" containerName="manager" Jul 06 19:56:58.738424 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.738408 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.740953 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.740928 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-173a5950\"/\"e2e-lifecycle-model-173a5950-kserve-self-signed-certs\"" Jul 06 19:56:58.741663 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.741646 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-173a5950\"/\"default-dockercfg-m2hhh\"" Jul 06 19:56:58.741819 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.741808 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-173a5950\"/\"kube-root-ca.crt\"" Jul 06 19:56:58.741885 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.741868 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-173a5950\"/\"openshift-service-ca.crt\"" Jul 06 19:56:58.759079 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.759054 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8"] Jul 06 19:56:58.843293 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.843262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-dshm\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.843293 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.843295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-home\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.843474 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.843313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-model-cache\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.843474 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.843332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tls-certs\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.843474 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.843369 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tmp-dir\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.843474 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.843396 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt9wq\" (UniqueName: \"kubernetes.io/projected/e40caa25-0d3e-49a0-95d5-f7b4b3742372-kube-api-access-xt9wq\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.944052 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.944020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-dshm\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.944052 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.944055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-home\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.944283 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.944076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-model-cache\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.944283 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.944095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tls-certs\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.944283 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.944122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tmp-dir\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.944283 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.944160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt9wq\" (UniqueName: \"kubernetes.io/projected/e40caa25-0d3e-49a0-95d5-f7b4b3742372-kube-api-access-xt9wq\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.944627 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.944590 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-home\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.944736 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.944590 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-model-cache\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.944736 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.944648 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tmp-dir\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.946531 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.946507 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-dshm\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.946636 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.946614 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tls-certs\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:58.955875 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:58.955848 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt9wq\" (UniqueName: \"kubernetes.io/projected/e40caa25-0d3e-49a0-95d5-f7b4b3742372-kube-api-access-xt9wq\") pod \"e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:59.054446 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:59.054375 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:56:59.196766 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:59.196735 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8"] Jul 06 19:56:59.198068 ip-10-0-140-103 kubenswrapper[2573]: W0706 19:56:59.198037 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode40caa25_0d3e_49a0_95d5_f7b4b3742372.slice/crio-5cc6063f1ebcbeb8d433caf5dd599de33514854e99fb93f6bba47e3cb3e3fc66 WatchSource:0}: Error finding container 5cc6063f1ebcbeb8d433caf5dd599de33514854e99fb93f6bba47e3cb3e3fc66: Status 404 returned error can't find the container with id 5cc6063f1ebcbeb8d433caf5dd599de33514854e99fb93f6bba47e3cb3e3fc66 Jul 06 19:56:59.414688 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:59.414657 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" event={"ID":"e40caa25-0d3e-49a0-95d5-f7b4b3742372","Type":"ContainerStarted","Data":"572f142922e1e1db6c9d7bc18afdf94a4e86c78fa3a9331bd615277206b9cabc"} Jul 06 19:56:59.414852 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:59.414697 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" event={"ID":"e40caa25-0d3e-49a0-95d5-f7b4b3742372","Type":"ContainerStarted","Data":"5cc6063f1ebcbeb8d433caf5dd599de33514854e99fb93f6bba47e3cb3e3fc66"} Jul 06 19:56:59.440393 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:56:59.440343 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" podStartSLOduration=1.440308949 podStartE2EDuration="1.440308949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:59.438557938 +0000 UTC m=+2043.535785693" watchObservedRunningTime="2026-07-06 19:56:59.440308949 +0000 UTC m=+2043.537536714" Jul 06 19:57:09.055394 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:09.055359 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:57:09.055803 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:09.055447 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:57:09.067867 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:09.067843 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:57:09.475013 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:09.474983 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:57:33.516654 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:33.516576 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8"] Jul 06 19:57:48.541403 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.541366 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" podUID="e40caa25-0d3e-49a0-95d5-f7b4b3742372" containerName="main" containerID="cri-o://572f142922e1e1db6c9d7bc18afdf94a4e86c78fa3a9331bd615277206b9cabc" gracePeriod=45 Jul 06 19:57:48.662577 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.662547 2573 generic.go:356] "Generic (PLEG): container finished" podID="e40caa25-0d3e-49a0-95d5-f7b4b3742372" containerID="572f142922e1e1db6c9d7bc18afdf94a4e86c78fa3a9331bd615277206b9cabc" exitCode=0 Jul 06 19:57:48.662684 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.662571 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" event={"ID":"e40caa25-0d3e-49a0-95d5-f7b4b3742372","Type":"ContainerDied","Data":"572f142922e1e1db6c9d7bc18afdf94a4e86c78fa3a9331bd615277206b9cabc"} Jul 06 19:57:48.793863 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.793807 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:57:48.897469 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897442 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-model-cache\") pod \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " Jul 06 19:57:48.897624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897492 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-dshm\") pod \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " Jul 06 19:57:48.897624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897509 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-home\") pod \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " Jul 06 19:57:48.897624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897531 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tls-certs\") pod \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " Jul 06 19:57:48.897624 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897594 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tmp-dir\") pod \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " Jul 06 19:57:48.897816 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897644 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt9wq\" (UniqueName: \"kubernetes.io/projected/e40caa25-0d3e-49a0-95d5-f7b4b3742372-kube-api-access-xt9wq\") pod \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\" (UID: \"e40caa25-0d3e-49a0-95d5-f7b4b3742372\") " Jul 06 19:57:48.897816 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897692 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-model-cache" (OuterVolumeSpecName: "model-cache") pod "e40caa25-0d3e-49a0-95d5-f7b4b3742372" (UID: "e40caa25-0d3e-49a0-95d5-f7b4b3742372"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:48.897899 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897815 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-home" (OuterVolumeSpecName: "home") pod "e40caa25-0d3e-49a0-95d5-f7b4b3742372" (UID: "e40caa25-0d3e-49a0-95d5-f7b4b3742372"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:48.897937 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897893 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e40caa25-0d3e-49a0-95d5-f7b4b3742372" (UID: "e40caa25-0d3e-49a0-95d5-f7b4b3742372"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:48.898047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897965 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:57:48.898047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897980 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:57:48.898047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.897989 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:57:48.899686 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.899656 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-dshm" (OuterVolumeSpecName: "dshm") pod "e40caa25-0d3e-49a0-95d5-f7b4b3742372" (UID: "e40caa25-0d3e-49a0-95d5-f7b4b3742372"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:48.899794 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.899743 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e40caa25-0d3e-49a0-95d5-f7b4b3742372" (UID: "e40caa25-0d3e-49a0-95d5-f7b4b3742372"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:57:48.899794 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.899756 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40caa25-0d3e-49a0-95d5-f7b4b3742372-kube-api-access-xt9wq" (OuterVolumeSpecName: "kube-api-access-xt9wq") pod "e40caa25-0d3e-49a0-95d5-f7b4b3742372" (UID: "e40caa25-0d3e-49a0-95d5-f7b4b3742372"). InnerVolumeSpecName "kube-api-access-xt9wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:57:48.998867 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.998835 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt9wq\" (UniqueName: \"kubernetes.io/projected/e40caa25-0d3e-49a0-95d5-f7b4b3742372-kube-api-access-xt9wq\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:57:48.998867 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.998860 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40caa25-0d3e-49a0-95d5-f7b4b3742372-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:57:48.998867 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:48.998871 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40caa25-0d3e-49a0-95d5-f7b4b3742372-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 19:57:49.668448 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:49.668421 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" Jul 06 19:57:49.668906 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:49.668465 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8" event={"ID":"e40caa25-0d3e-49a0-95d5-f7b4b3742372","Type":"ContainerDied","Data":"5cc6063f1ebcbeb8d433caf5dd599de33514854e99fb93f6bba47e3cb3e3fc66"} Jul 06 19:57:49.668906 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:49.668523 2573 scope.go:117] "RemoveContainer" containerID="572f142922e1e1db6c9d7bc18afdf94a4e86c78fa3a9331bd615277206b9cabc" Jul 06 19:57:49.690722 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:49.690695 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8"] Jul 06 19:57:49.703628 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:49.699117 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-173a5950/e2e-lifecycle-model-173a5950-kserve-7dbbcbfd4c-q59q8"] Jul 06 19:57:50.394047 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:50.394012 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e40caa25-0d3e-49a0-95d5-f7b4b3742372" path="/var/lib/kubelet/pods/e40caa25-0d3e-49a0-95d5-f7b4b3742372/volumes" Jul 06 19:57:56.451820 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:56.451775 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 19:57:56.459948 ip-10-0-140-103 kubenswrapper[2573]: I0706 19:57:56.459923 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 20:00:00.143380 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.143337 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-5ggls"] Jul 06 20:00:00.143878 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.143693 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e40caa25-0d3e-49a0-95d5-f7b4b3742372" containerName="main" Jul 06 20:00:00.143878 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.143705 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e40caa25-0d3e-49a0-95d5-f7b4b3742372" containerName="main" Jul 06 20:00:00.143878 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.143788 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e40caa25-0d3e-49a0-95d5-f7b4b3742372" containerName="main" Jul 06 20:00:00.146443 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.146413 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm"] Jul 06 20:00:00.146585 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.146526 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" Jul 06 20:00:00.149143 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.149119 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vwbwq\"" Jul 06 20:00:00.149804 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.149782 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4"] Jul 06 20:00:00.149912 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.149869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" Jul 06 20:00:00.153195 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.153171 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" Jul 06 20:00:00.157148 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.156432 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm"] Jul 06 20:00:00.160384 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.160347 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4"] Jul 06 20:00:00.162979 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.162953 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-5ggls"] Jul 06 20:00:00.219974 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.219936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgp7h\" (UniqueName: \"kubernetes.io/projected/965fba7f-c234-41d4-b3ce-5f93d1f84bd4-kube-api-access-xgp7h\") pod \"maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4\" (UID: \"965fba7f-c234-41d4-b3ce-5f93d1f84bd4\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" Jul 06 20:00:00.220153 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.220006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qf6\" (UniqueName: \"kubernetes.io/projected/c30feacb-c3a9-4c89-b856-e31cd5351fed-kube-api-access-j2qf6\") pod \"maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm\" (UID: \"c30feacb-c3a9-4c89-b856-e31cd5351fed\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" Jul 06 20:00:00.220153 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.220031 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6qz\" (UniqueName: \"kubernetes.io/projected/c3659e28-327f-48b5-9732-4bff9d8464c3-kube-api-access-fj6qz\") pod \"maas-api-key-cleanup-29722800-5ggls\" (UID: \"c3659e28-327f-48b5-9732-4bff9d8464c3\") " pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" Jul 06 20:00:00.321281 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.321240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgp7h\" (UniqueName: \"kubernetes.io/projected/965fba7f-c234-41d4-b3ce-5f93d1f84bd4-kube-api-access-xgp7h\") pod \"maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4\" (UID: \"965fba7f-c234-41d4-b3ce-5f93d1f84bd4\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" Jul 06 20:00:00.321466 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.321327 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qf6\" (UniqueName: \"kubernetes.io/projected/c30feacb-c3a9-4c89-b856-e31cd5351fed-kube-api-access-j2qf6\") pod \"maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm\" (UID: \"c30feacb-c3a9-4c89-b856-e31cd5351fed\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" Jul 06 20:00:00.321466 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.321353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6qz\" (UniqueName: \"kubernetes.io/projected/c3659e28-327f-48b5-9732-4bff9d8464c3-kube-api-access-fj6qz\") pod \"maas-api-key-cleanup-29722800-5ggls\" (UID: \"c3659e28-327f-48b5-9732-4bff9d8464c3\") " pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" Jul 06 20:00:00.330296 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.330261 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qf6\" (UniqueName: \"kubernetes.io/projected/c30feacb-c3a9-4c89-b856-e31cd5351fed-kube-api-access-j2qf6\") pod \"maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm\" (UID: \"c30feacb-c3a9-4c89-b856-e31cd5351fed\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" Jul 06 20:00:00.330296 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.330261 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6qz\" (UniqueName: \"kubernetes.io/projected/c3659e28-327f-48b5-9732-4bff9d8464c3-kube-api-access-fj6qz\") pod \"maas-api-key-cleanup-29722800-5ggls\" (UID: \"c3659e28-327f-48b5-9732-4bff9d8464c3\") " pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" Jul 06 20:00:00.330475 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.330332 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgp7h\" (UniqueName: \"kubernetes.io/projected/965fba7f-c234-41d4-b3ce-5f93d1f84bd4-kube-api-access-xgp7h\") pod \"maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4\" (UID: \"965fba7f-c234-41d4-b3ce-5f93d1f84bd4\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" Jul 06 20:00:00.460739 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.460641 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" Jul 06 20:00:00.471058 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.471033 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" Jul 06 20:00:00.477354 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.477312 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" Jul 06 20:00:00.834389 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.834360 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-5ggls"] Jul 06 20:00:00.835820 ip-10-0-140-103 kubenswrapper[2573]: W0706 20:00:00.835793 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3659e28_327f_48b5_9732_4bff9d8464c3.slice/crio-3066819eb95c01c2d44f800f011803afd2201bfcd78be7c36744cec5ec4984d2 WatchSource:0}: Error finding container 3066819eb95c01c2d44f800f011803afd2201bfcd78be7c36744cec5ec4984d2: Status 404 returned error can't find the container with id 3066819eb95c01c2d44f800f011803afd2201bfcd78be7c36744cec5ec4984d2 Jul 06 20:00:00.837948 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.837924 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:00:00.864232 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.864117 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4"] Jul 06 20:00:00.867200 ip-10-0-140-103 kubenswrapper[2573]: W0706 20:00:00.867168 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965fba7f_c234_41d4_b3ce_5f93d1f84bd4.slice/crio-b81727ff3f90fc25b2c34f342a700c5e8e8390b09425bc179b947d6170db5009 WatchSource:0}: Error finding container b81727ff3f90fc25b2c34f342a700c5e8e8390b09425bc179b947d6170db5009: Status 404 returned error can't find the container with id b81727ff3f90fc25b2c34f342a700c5e8e8390b09425bc179b947d6170db5009 Jul 06 20:00:00.867595 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:00.867564 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm"] Jul 06 20:00:00.868440 ip-10-0-140-103 kubenswrapper[2573]: W0706 20:00:00.868419 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30feacb_c3a9_4c89_b856_e31cd5351fed.slice/crio-87a255cb3e83927fc9a8649f39a78313da9376c35ca023fd38a1b92763c28760 WatchSource:0}: Error finding container 87a255cb3e83927fc9a8649f39a78313da9376c35ca023fd38a1b92763c28760: Status 404 returned error can't find the container with id 87a255cb3e83927fc9a8649f39a78313da9376c35ca023fd38a1b92763c28760 Jul 06 20:00:01.350430 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:01.350396 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" event={"ID":"965fba7f-c234-41d4-b3ce-5f93d1f84bd4","Type":"ContainerStarted","Data":"b81727ff3f90fc25b2c34f342a700c5e8e8390b09425bc179b947d6170db5009"} Jul 06 20:00:01.351603 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:01.351580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" event={"ID":"c3659e28-327f-48b5-9732-4bff9d8464c3","Type":"ContainerStarted","Data":"3066819eb95c01c2d44f800f011803afd2201bfcd78be7c36744cec5ec4984d2"} Jul 06 20:00:01.352730 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:01.352709 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" event={"ID":"c30feacb-c3a9-4c89-b856-e31cd5351fed","Type":"ContainerStarted","Data":"87a255cb3e83927fc9a8649f39a78313da9376c35ca023fd38a1b92763c28760"} Jul 06 20:00:08.393439 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:08.393397 2573 generic.go:356] "Generic (PLEG): container finished" podID="c30feacb-c3a9-4c89-b856-e31cd5351fed" containerID="62d7ca54b60ea53f1d53a79e33dc552bfa0b79655b7e11166b5d57e88b2a6514" exitCode=0 Jul 06 20:00:08.394275 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:08.394251 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" event={"ID":"c30feacb-c3a9-4c89-b856-e31cd5351fed","Type":"ContainerDied","Data":"62d7ca54b60ea53f1d53a79e33dc552bfa0b79655b7e11166b5d57e88b2a6514"} Jul 06 20:00:09.536109 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:09.536079 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" Jul 06 20:00:09.610195 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:09.610151 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2qf6\" (UniqueName: \"kubernetes.io/projected/c30feacb-c3a9-4c89-b856-e31cd5351fed-kube-api-access-j2qf6\") pod \"c30feacb-c3a9-4c89-b856-e31cd5351fed\" (UID: \"c30feacb-c3a9-4c89-b856-e31cd5351fed\") " Jul 06 20:00:09.612388 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:09.612363 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30feacb-c3a9-4c89-b856-e31cd5351fed-kube-api-access-j2qf6" (OuterVolumeSpecName: "kube-api-access-j2qf6") pod "c30feacb-c3a9-4c89-b856-e31cd5351fed" (UID: "c30feacb-c3a9-4c89-b856-e31cd5351fed"). InnerVolumeSpecName "kube-api-access-j2qf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:09.711898 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:09.711811 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2qf6\" (UniqueName: \"kubernetes.io/projected/c30feacb-c3a9-4c89-b856-e31cd5351fed-kube-api-access-j2qf6\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:00:10.404399 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:10.404372 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" Jul 06 20:00:10.404552 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:10.404407 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm" event={"ID":"c30feacb-c3a9-4c89-b856-e31cd5351fed","Type":"ContainerDied","Data":"87a255cb3e83927fc9a8649f39a78313da9376c35ca023fd38a1b92763c28760"} Jul 06 20:00:10.404552 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:10.404458 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87a255cb3e83927fc9a8649f39a78313da9376c35ca023fd38a1b92763c28760" Jul 06 20:00:15.995502 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:15.995465 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5"] Jul 06 20:00:15.995920 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:15.995815 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30feacb-c3a9-4c89-b856-e31cd5351fed" containerName="cleanup" Jul 06 20:00:15.995920 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:15.995825 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30feacb-c3a9-4c89-b856-e31cd5351fed" containerName="cleanup" Jul 06 20:00:15.995920 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:15.995897 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30feacb-c3a9-4c89-b856-e31cd5351fed" containerName="cleanup" Jul 06 20:00:15.998998 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:15.998979 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.001446 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.001420 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6bd627\"/\"default-dockercfg-9kpm7\"" Jul 06 20:00:16.002231 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.002190 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6bd627\"/\"kube-root-ca.crt\"" Jul 06 20:00:16.002355 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.002240 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6bd627\"/\"openshift-service-ca.crt\"" Jul 06 20:00:16.002355 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.002245 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6bd627\"/\"test-model-6bd627-kserve-self-signed-certs\"" Jul 06 20:00:16.011787 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.011751 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5"] Jul 06 20:00:16.069498 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.069445 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-kube-api-access-58xkl\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.069697 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.069517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-model-cache\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.069697 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.069555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-dshm\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.069697 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.069578 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tmp-dir\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.069697 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.069608 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tls-certs\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.069841 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.069734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-home\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.170289 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.170236 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-kube-api-access-58xkl\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.170507 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.170296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-model-cache\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.170507 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.170346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-dshm\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.170507 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.170374 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tmp-dir\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.170507 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.170406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tls-certs\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.170712 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.170532 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-home\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.171068 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.171025 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-model-cache\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.171293 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.171263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-home\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.171482 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.171457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tmp-dir\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.173306 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.173284 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tls-certs\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.173640 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.173613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-dshm\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.182448 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.182400 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-kube-api-access-58xkl\") pod \"test-model-6bd627-kserve-777ddcd867-p6fl5\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.311037 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.310943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:16.458482 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:16.458456 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5"] Jul 06 20:00:16.460222 ip-10-0-140-103 kubenswrapper[2573]: W0706 20:00:16.460178 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cbe05a3_d0d4_4e8b_a436_c35a005b3832.slice/crio-a2b8e2c935afe74ccfdc3062fe3bf1f9465adfdb361bc4695a82bedc821e3a5a WatchSource:0}: Error finding container a2b8e2c935afe74ccfdc3062fe3bf1f9465adfdb361bc4695a82bedc821e3a5a: Status 404 returned error can't find the container with id a2b8e2c935afe74ccfdc3062fe3bf1f9465adfdb361bc4695a82bedc821e3a5a Jul 06 20:00:17.447960 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:17.447923 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" event={"ID":"c3659e28-327f-48b5-9732-4bff9d8464c3","Type":"ContainerStarted","Data":"cbf05aa1e2991f78f59cd85a36a24fd629f5a08c0453aee02cbc839a5f9a09d6"} Jul 06 20:00:17.449435 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:17.449406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" event={"ID":"8cbe05a3-d0d4-4e8b-a436-c35a005b3832","Type":"ContainerStarted","Data":"3ece42ce739cec3a51f775399546f33b2a06bf77767c140a9ae4542411dedddd"} Jul 06 20:00:17.449562 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:17.449440 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" event={"ID":"8cbe05a3-d0d4-4e8b-a436-c35a005b3832","Type":"ContainerStarted","Data":"a2b8e2c935afe74ccfdc3062fe3bf1f9465adfdb361bc4695a82bedc821e3a5a"} Jul 06 20:00:17.494510 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:17.494463 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" podStartSLOduration=1.72794476 podStartE2EDuration="17.494447882s" totalImagesPullingTime="15.766503122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:17.493020513 +0000 UTC m=+2241.590248267" watchObservedRunningTime="2026-07-06 20:00:17.494447882 +0000 UTC m=+2241.591675636" Jul 06 20:00:17.523566 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:17.523517 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" podStartSLOduration=2.523501731 podStartE2EDuration="2.523501731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:17.521337143 +0000 UTC m=+2241.618564909" watchObservedRunningTime="2026-07-06 20:00:17.523501731 +0000 UTC m=+2241.620729486" Jul 06 20:00:18.457325 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:18.457235 2573 generic.go:356] "Generic (PLEG): container finished" podID="c3659e28-327f-48b5-9732-4bff9d8464c3" containerID="cbf05aa1e2991f78f59cd85a36a24fd629f5a08c0453aee02cbc839a5f9a09d6" exitCode=0 Jul 06 20:00:18.457325 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:18.457304 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" event={"ID":"c3659e28-327f-48b5-9732-4bff9d8464c3","Type":"ContainerDied","Data":"cbf05aa1e2991f78f59cd85a36a24fd629f5a08c0453aee02cbc839a5f9a09d6"} Jul 06 20:00:19.595455 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:19.595422 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" Jul 06 20:00:19.706547 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:19.706507 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj6qz\" (UniqueName: \"kubernetes.io/projected/c3659e28-327f-48b5-9732-4bff9d8464c3-kube-api-access-fj6qz\") pod \"c3659e28-327f-48b5-9732-4bff9d8464c3\" (UID: \"c3659e28-327f-48b5-9732-4bff9d8464c3\") " Jul 06 20:00:19.708652 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:19.708622 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3659e28-327f-48b5-9732-4bff9d8464c3-kube-api-access-fj6qz" (OuterVolumeSpecName: "kube-api-access-fj6qz") pod "c3659e28-327f-48b5-9732-4bff9d8464c3" (UID: "c3659e28-327f-48b5-9732-4bff9d8464c3"). InnerVolumeSpecName "kube-api-access-fj6qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:19.807909 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:19.807874 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj6qz\" (UniqueName: \"kubernetes.io/projected/c3659e28-327f-48b5-9732-4bff9d8464c3-kube-api-access-fj6qz\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:00:20.476021 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:20.475980 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" event={"ID":"c3659e28-327f-48b5-9732-4bff9d8464c3","Type":"ContainerDied","Data":"3066819eb95c01c2d44f800f011803afd2201bfcd78be7c36744cec5ec4984d2"} Jul 06 20:00:20.476184 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:20.476058 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3066819eb95c01c2d44f800f011803afd2201bfcd78be7c36744cec5ec4984d2" Jul 06 20:00:20.476184 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:20.475981 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-5ggls" Jul 06 20:00:22.488902 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:22.488869 2573 generic.go:356] "Generic (PLEG): container finished" podID="965fba7f-c234-41d4-b3ce-5f93d1f84bd4" containerID="eadb4976f33b3b04fda604beb88570d48b42d09baf0bbc9a86fcb281dd05663b" exitCode=0 Jul 06 20:00:22.489311 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:22.488922 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" event={"ID":"965fba7f-c234-41d4-b3ce-5f93d1f84bd4","Type":"ContainerDied","Data":"eadb4976f33b3b04fda604beb88570d48b42d09baf0bbc9a86fcb281dd05663b"} Jul 06 20:00:23.631443 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:23.631412 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" Jul 06 20:00:23.744270 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:23.744228 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgp7h\" (UniqueName: \"kubernetes.io/projected/965fba7f-c234-41d4-b3ce-5f93d1f84bd4-kube-api-access-xgp7h\") pod \"965fba7f-c234-41d4-b3ce-5f93d1f84bd4\" (UID: \"965fba7f-c234-41d4-b3ce-5f93d1f84bd4\") " Jul 06 20:00:23.746455 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:23.746394 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965fba7f-c234-41d4-b3ce-5f93d1f84bd4-kube-api-access-xgp7h" (OuterVolumeSpecName: "kube-api-access-xgp7h") pod "965fba7f-c234-41d4-b3ce-5f93d1f84bd4" (UID: "965fba7f-c234-41d4-b3ce-5f93d1f84bd4"). InnerVolumeSpecName "kube-api-access-xgp7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:23.845382 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:23.845345 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgp7h\" (UniqueName: \"kubernetes.io/projected/965fba7f-c234-41d4-b3ce-5f93d1f84bd4-kube-api-access-xgp7h\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:00:24.500667 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:24.500628 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" event={"ID":"965fba7f-c234-41d4-b3ce-5f93d1f84bd4","Type":"ContainerDied","Data":"b81727ff3f90fc25b2c34f342a700c5e8e8390b09425bc179b947d6170db5009"} Jul 06 20:00:24.500843 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:24.500676 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4" Jul 06 20:00:24.500843 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:24.500683 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81727ff3f90fc25b2c34f342a700c5e8e8390b09425bc179b947d6170db5009" Jul 06 20:00:26.311496 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:26.311454 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:26.311910 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:26.311514 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:26.324502 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:26.324469 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:26.523089 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:26.523059 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:00:51.532146 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:51.532113 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5"] Jul 06 20:00:52.447159 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:52.447116 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm"] Jul 06 20:00:52.452760 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:52.452725 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-34eebf-29722800-qsqgm"] Jul 06 20:00:54.394366 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:00:54.394321 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c30feacb-c3a9-4c89-b856-e31cd5351fed" path="/var/lib/kubelet/pods/c30feacb-c3a9-4c89-b856-e31cd5351fed/volumes" Jul 06 20:01:06.559548 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.559481 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" podUID="8cbe05a3-d0d4-4e8b-a436-c35a005b3832" containerName="main" containerID="cri-o://3ece42ce739cec3a51f775399546f33b2a06bf77767c140a9ae4542411dedddd" gracePeriod=45 Jul 06 20:01:06.726780 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.726733 2573 generic.go:356] "Generic (PLEG): container finished" podID="8cbe05a3-d0d4-4e8b-a436-c35a005b3832" containerID="3ece42ce739cec3a51f775399546f33b2a06bf77767c140a9ae4542411dedddd" exitCode=0 Jul 06 20:01:06.726950 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.726805 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" event={"ID":"8cbe05a3-d0d4-4e8b-a436-c35a005b3832","Type":"ContainerDied","Data":"3ece42ce739cec3a51f775399546f33b2a06bf77767c140a9ae4542411dedddd"} Jul 06 20:01:06.821572 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.821510 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:01:06.943414 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.943381 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-kube-api-access-58xkl\") pod \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " Jul 06 20:01:06.943632 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.943439 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-dshm\") pod \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " Jul 06 20:01:06.943632 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.943472 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tmp-dir\") pod \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " Jul 06 20:01:06.943774 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.943753 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-model-cache\") pod \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " Jul 06 20:01:06.943829 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.943785 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tls-certs\") pod \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " Jul 06 20:01:06.943887 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.943828 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8cbe05a3-d0d4-4e8b-a436-c35a005b3832" (UID: "8cbe05a3-d0d4-4e8b-a436-c35a005b3832"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:06.943946 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.943887 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-home\") pod \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\" (UID: \"8cbe05a3-d0d4-4e8b-a436-c35a005b3832\") " Jul 06 20:01:06.944122 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.944064 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-model-cache" (OuterVolumeSpecName: "model-cache") pod "8cbe05a3-d0d4-4e8b-a436-c35a005b3832" (UID: "8cbe05a3-d0d4-4e8b-a436-c35a005b3832"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:06.944420 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.944396 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-home" (OuterVolumeSpecName: "home") pod "8cbe05a3-d0d4-4e8b-a436-c35a005b3832" (UID: "8cbe05a3-d0d4-4e8b-a436-c35a005b3832"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:06.944544 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.944506 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:01:06.944544 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.944520 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:01:06.944544 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.944529 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:01:06.946278 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.946247 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-dshm" (OuterVolumeSpecName: "dshm") pod "8cbe05a3-d0d4-4e8b-a436-c35a005b3832" (UID: "8cbe05a3-d0d4-4e8b-a436-c35a005b3832"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:06.946278 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.946266 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-kube-api-access-58xkl" (OuterVolumeSpecName: "kube-api-access-58xkl") pod "8cbe05a3-d0d4-4e8b-a436-c35a005b3832" (UID: "8cbe05a3-d0d4-4e8b-a436-c35a005b3832"). InnerVolumeSpecName "kube-api-access-58xkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:06.946639 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:06.946614 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8cbe05a3-d0d4-4e8b-a436-c35a005b3832" (UID: "8cbe05a3-d0d4-4e8b-a436-c35a005b3832"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:01:07.045419 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:07.045372 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:01:07.045419 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:07.045407 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-kube-api-access-58xkl\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:01:07.045419 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:07.045417 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbe05a3-d0d4-4e8b-a436-c35a005b3832-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:01:07.733904 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:07.733874 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" Jul 06 20:01:07.733904 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:07.733874 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5" event={"ID":"8cbe05a3-d0d4-4e8b-a436-c35a005b3832","Type":"ContainerDied","Data":"a2b8e2c935afe74ccfdc3062fe3bf1f9465adfdb361bc4695a82bedc821e3a5a"} Jul 06 20:01:07.734430 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:07.733934 2573 scope.go:117] "RemoveContainer" containerID="3ece42ce739cec3a51f775399546f33b2a06bf77767c140a9ae4542411dedddd" Jul 06 20:01:07.755455 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:07.755423 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5"] Jul 06 20:01:07.759098 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:07.759068 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6bd627/test-model-6bd627-kserve-777ddcd867-p6fl5"] Jul 06 20:01:08.394283 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:08.394250 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cbe05a3-d0d4-4e8b-a436-c35a005b3832" path="/var/lib/kubelet/pods/8cbe05a3-d0d4-4e8b-a436-c35a005b3832/volumes" Jul 06 20:01:35.707127 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:35.707092 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4"] Jul 06 20:01:35.710407 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:35.710375 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-6bd627-29722800-4v9z4"] Jul 06 20:01:36.395196 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:01:36.395163 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="965fba7f-c234-41d4-b3ce-5f93d1f84bd4" path="/var/lib/kubelet/pods/965fba7f-c234-41d4-b3ce-5f93d1f84bd4/volumes" Jul 06 20:02:54.088402 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088362 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds"] Jul 06 20:02:54.088867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088728 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965fba7f-c234-41d4-b3ce-5f93d1f84bd4" containerName="cleanup" Jul 06 20:02:54.088867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088740 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="965fba7f-c234-41d4-b3ce-5f93d1f84bd4" containerName="cleanup" Jul 06 20:02:54.088867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088757 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cbe05a3-d0d4-4e8b-a436-c35a005b3832" containerName="main" Jul 06 20:02:54.088867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088764 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cbe05a3-d0d4-4e8b-a436-c35a005b3832" containerName="main" Jul 06 20:02:54.088867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088774 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3659e28-327f-48b5-9732-4bff9d8464c3" containerName="cleanup" Jul 06 20:02:54.088867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088780 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3659e28-327f-48b5-9732-4bff9d8464c3" containerName="cleanup" Jul 06 20:02:54.088867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088845 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="965fba7f-c234-41d4-b3ce-5f93d1f84bd4" containerName="cleanup" Jul 06 20:02:54.088867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088853 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3659e28-327f-48b5-9732-4bff9d8464c3" containerName="cleanup" Jul 06 20:02:54.088867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.088861 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cbe05a3-d0d4-4e8b-a436-c35a005b3832" containerName="main" Jul 06 20:02:54.092544 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.092527 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.096055 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.096026 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c21a78\"/\"kube-root-ca.crt\"" Jul 06 20:02:54.096189 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.096058 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c21a78\"/\"auth-test-model-c21a78-kserve-self-signed-certs\"" Jul 06 20:02:54.100719 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.100696 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c21a78\"/\"openshift-service-ca.crt\"" Jul 06 20:02:54.100851 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.100833 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c21a78\"/\"default-dockercfg-9nxz5\"" Jul 06 20:02:54.129079 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.129047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-model-cache\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.129295 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.129103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tmp-dir\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.129295 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.129168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzkp\" (UniqueName: \"kubernetes.io/projected/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-kube-api-access-qjzkp\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.129295 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.129248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-dshm\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.129295 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.129290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tls-certs\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.129639 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.129355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-home\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.230573 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.230537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tmp-dir\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.230573 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.230578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzkp\" (UniqueName: \"kubernetes.io/projected/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-kube-api-access-qjzkp\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.230851 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.230719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-dshm\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.230851 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.230785 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tls-certs\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.230965 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.230866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-home\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.230965 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.230903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-model-cache\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.230965 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.230934 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tmp-dir\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.231121 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.231100 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-home\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.231312 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.231291 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-model-cache\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.232942 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.232922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-dshm\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.233181 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.233163 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tls-certs\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:54.891180 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:54.891140 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds"] Jul 06 20:02:55.297358 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:55.297272 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzkp\" (UniqueName: \"kubernetes.io/projected/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-kube-api-access-qjzkp\") pod \"auth-test-model-c21a78-kserve-695cff5ff9-z2hds\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:55.301959 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:55.301895 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:02:55.449325 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:55.449291 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds"] Jul 06 20:02:55.452649 ip-10-0-140-103 kubenswrapper[2573]: W0706 20:02:55.452621 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9f66d85_eaef_4a61_8af1_5e7ddd7b47e8.slice/crio-bbca7b59c95a6adc01093f6e3b333f11e2e0ed1dd2a26b8bef2254ffa06d76ee WatchSource:0}: Error finding container bbca7b59c95a6adc01093f6e3b333f11e2e0ed1dd2a26b8bef2254ffa06d76ee: Status 404 returned error can't find the container with id bbca7b59c95a6adc01093f6e3b333f11e2e0ed1dd2a26b8bef2254ffa06d76ee Jul 06 20:02:56.333087 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:56.333045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" event={"ID":"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8","Type":"ContainerStarted","Data":"d87fadfb5ab6565cf777baf7cf0c7e05cd1a7342025ac1d4d19d3fef0a0d102a"} Jul 06 20:02:56.333087 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:56.333094 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" event={"ID":"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8","Type":"ContainerStarted","Data":"bbca7b59c95a6adc01093f6e3b333f11e2e0ed1dd2a26b8bef2254ffa06d76ee"} Jul 06 20:02:56.352682 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:56.352625 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" podStartSLOduration=5.352609191 podStartE2EDuration="5.352609191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:02:56.3504963 +0000 UTC m=+2400.447724053" watchObservedRunningTime="2026-07-06 20:02:56.352609191 +0000 UTC m=+2400.449836945" Jul 06 20:02:56.491344 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:56.491228 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 20:02:56.509556 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:02:56.495136 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 20:03:05.302693 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:03:05.302653 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:03:05.303113 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:03:05.302930 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:03:05.315184 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:03:05.315153 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:03:05.393534 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:03:05.393503 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:05:19.024599 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:19.024511 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds"] Jul 06 20:05:24.064729 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.064687 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn"] Jul 06 20:05:24.067593 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.067575 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.085725 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.085699 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c21a78\"/\"sub-test-model-c21a78-kserve-self-signed-certs\"" Jul 06 20:05:24.098508 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.098474 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn"] Jul 06 20:05:24.106112 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.106090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609a0174-08f0-48a4-965d-02b0df4f4cd2-tls-certs\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.106239 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.106138 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-dshm\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.106239 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.106160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-model-cache\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.106314 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.106246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-home\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.106314 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.106279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97pf\" (UniqueName: \"kubernetes.io/projected/609a0174-08f0-48a4-965d-02b0df4f4cd2-kube-api-access-j97pf\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.106383 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.106374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-tmp-dir\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.207574 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.207533 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-tmp-dir\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.207574 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.207575 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609a0174-08f0-48a4-965d-02b0df4f4cd2-tls-certs\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.207813 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.207608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-dshm\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.207813 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.207626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-model-cache\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.207813 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.207657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-home\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.207813 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.207684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97pf\" (UniqueName: \"kubernetes.io/projected/609a0174-08f0-48a4-965d-02b0df4f4cd2-kube-api-access-j97pf\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.208021 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.207995 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-tmp-dir\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.208085 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.208039 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-model-cache\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.208144 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.208115 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-home\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.209996 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.209976 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-dshm\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.210138 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.210119 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609a0174-08f0-48a4-965d-02b0df4f4cd2-tls-certs\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.230388 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.230359 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97pf\" (UniqueName: \"kubernetes.io/projected/609a0174-08f0-48a4-965d-02b0df4f4cd2-kube-api-access-j97pf\") pod \"sub-test-model-c21a78-kserve-d55c88475-ktckn\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.377687 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.377646 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:24.531917 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:24.531874 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn"] Jul 06 20:05:24.539519 ip-10-0-140-103 kubenswrapper[2573]: W0706 20:05:24.539482 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod609a0174_08f0_48a4_965d_02b0df4f4cd2.slice/crio-8bc5397acda0fd5b08410f9ba693539ba49221a875e7cb8234ad3b793796d3a5 WatchSource:0}: Error finding container 8bc5397acda0fd5b08410f9ba693539ba49221a875e7cb8234ad3b793796d3a5: Status 404 returned error can't find the container with id 8bc5397acda0fd5b08410f9ba693539ba49221a875e7cb8234ad3b793796d3a5 Jul 06 20:05:25.143778 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:25.143733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" event={"ID":"609a0174-08f0-48a4-965d-02b0df4f4cd2","Type":"ContainerStarted","Data":"6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f"} Jul 06 20:05:25.144183 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:25.143788 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" event={"ID":"609a0174-08f0-48a4-965d-02b0df4f4cd2","Type":"ContainerStarted","Data":"8bc5397acda0fd5b08410f9ba693539ba49221a875e7cb8234ad3b793796d3a5"} Jul 06 20:05:25.168571 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:25.168509 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" podStartSLOduration=1.168491044 podStartE2EDuration="1.168491044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:05:25.166394496 +0000 UTC m=+2549.263622243" watchObservedRunningTime="2026-07-06 20:05:25.168491044 +0000 UTC m=+2549.265718799" Jul 06 20:05:34.044627 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.044563 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" podUID="c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" containerName="main" containerID="cri-o://d87fadfb5ab6565cf777baf7cf0c7e05cd1a7342025ac1d4d19d3fef0a0d102a" gracePeriod=45 Jul 06 20:05:34.193547 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.193516 2573 generic.go:356] "Generic (PLEG): container finished" podID="c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" containerID="d87fadfb5ab6565cf777baf7cf0c7e05cd1a7342025ac1d4d19d3fef0a0d102a" exitCode=0 Jul 06 20:05:34.193754 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.193571 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" event={"ID":"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8","Type":"ContainerDied","Data":"d87fadfb5ab6565cf777baf7cf0c7e05cd1a7342025ac1d4d19d3fef0a0d102a"} Jul 06 20:05:34.296502 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.296433 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:05:34.378304 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.378264 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:34.378483 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.378327 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:34.396170 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.396141 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:34.403774 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.403752 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tmp-dir\") pod \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " Jul 06 20:05:34.403869 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.403787 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjzkp\" (UniqueName: \"kubernetes.io/projected/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-kube-api-access-qjzkp\") pod \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " Jul 06 20:05:34.403869 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.403808 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tls-certs\") pod \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " Jul 06 20:05:34.403869 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.403846 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-home\") pod \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " Jul 06 20:05:34.403965 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.403876 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-dshm\") pod \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " Jul 06 20:05:34.403965 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.403945 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-model-cache\") pod \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\" (UID: \"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8\") " Jul 06 20:05:34.404108 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.404075 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" (UID: "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:34.404265 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.404170 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-home" (OuterVolumeSpecName: "home") pod "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" (UID: "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:34.404334 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.404302 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" (UID: "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:34.406048 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.406022 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-dshm" (OuterVolumeSpecName: "dshm") pod "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" (UID: "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:34.406048 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.406036 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" (UID: "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:05:34.406048 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.406043 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-kube-api-access-qjzkp" (OuterVolumeSpecName: "kube-api-access-qjzkp") pod "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" (UID: "c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8"). InnerVolumeSpecName "kube-api-access-qjzkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:05:34.505496 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.505455 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:05:34.505496 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.505487 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:05:34.505496 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.505496 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzkp\" (UniqueName: \"kubernetes.io/projected/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-kube-api-access-qjzkp\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:05:34.505496 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.505504 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:05:34.505767 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.505513 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:05:34.505767 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:34.505520 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:05:35.201088 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:35.201059 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" Jul 06 20:05:35.201591 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:35.201088 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds" event={"ID":"c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8","Type":"ContainerDied","Data":"bbca7b59c95a6adc01093f6e3b333f11e2e0ed1dd2a26b8bef2254ffa06d76ee"} Jul 06 20:05:35.201591 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:35.201134 2573 scope.go:117] "RemoveContainer" containerID="d87fadfb5ab6565cf777baf7cf0c7e05cd1a7342025ac1d4d19d3fef0a0d102a" Jul 06 20:05:35.213747 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:35.213724 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:05:35.223631 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:35.223602 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds"] Jul 06 20:05:35.231579 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:35.231550 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/auth-test-model-c21a78-kserve-695cff5ff9-z2hds"] Jul 06 20:05:36.394845 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:05:36.394815 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" path="/var/lib/kubelet/pods/c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8/volumes" Jul 06 20:06:25.955800 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:25.955717 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn"] Jul 06 20:06:40.975292 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:40.975231 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" podUID="609a0174-08f0-48a4-965d-02b0df4f4cd2" containerName="main" containerID="cri-o://6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f" gracePeriod=45 Jul 06 20:06:41.228927 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.228851 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:06:41.322441 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.322402 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609a0174-08f0-48a4-965d-02b0df4f4cd2-tls-certs\") pod \"609a0174-08f0-48a4-965d-02b0df4f4cd2\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " Jul 06 20:06:41.322441 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.322444 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j97pf\" (UniqueName: \"kubernetes.io/projected/609a0174-08f0-48a4-965d-02b0df4f4cd2-kube-api-access-j97pf\") pod \"609a0174-08f0-48a4-965d-02b0df4f4cd2\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " Jul 06 20:06:41.322695 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.322502 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-dshm\") pod \"609a0174-08f0-48a4-965d-02b0df4f4cd2\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " Jul 06 20:06:41.322695 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.322556 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-home\") pod \"609a0174-08f0-48a4-965d-02b0df4f4cd2\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " Jul 06 20:06:41.322695 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.322632 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-model-cache\") pod \"609a0174-08f0-48a4-965d-02b0df4f4cd2\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " Jul 06 20:06:41.322862 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.322696 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-tmp-dir\") pod \"609a0174-08f0-48a4-965d-02b0df4f4cd2\" (UID: \"609a0174-08f0-48a4-965d-02b0df4f4cd2\") " Jul 06 20:06:41.322862 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.322808 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-home" (OuterVolumeSpecName: "home") pod "609a0174-08f0-48a4-965d-02b0df4f4cd2" (UID: "609a0174-08f0-48a4-965d-02b0df4f4cd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:41.322966 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.322854 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "609a0174-08f0-48a4-965d-02b0df4f4cd2" (UID: "609a0174-08f0-48a4-965d-02b0df4f4cd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:41.322966 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.322953 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "609a0174-08f0-48a4-965d-02b0df4f4cd2" (UID: "609a0174-08f0-48a4-965d-02b0df4f4cd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:41.323077 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.323064 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:06:41.323138 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.323082 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:06:41.323138 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.323094 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:06:41.324802 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.324770 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-dshm" (OuterVolumeSpecName: "dshm") pod "609a0174-08f0-48a4-965d-02b0df4f4cd2" (UID: "609a0174-08f0-48a4-965d-02b0df4f4cd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:41.324802 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.324771 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609a0174-08f0-48a4-965d-02b0df4f4cd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "609a0174-08f0-48a4-965d-02b0df4f4cd2" (UID: "609a0174-08f0-48a4-965d-02b0df4f4cd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:06:41.324990 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.324868 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609a0174-08f0-48a4-965d-02b0df4f4cd2-kube-api-access-j97pf" (OuterVolumeSpecName: "kube-api-access-j97pf") pod "609a0174-08f0-48a4-965d-02b0df4f4cd2" (UID: "609a0174-08f0-48a4-965d-02b0df4f4cd2"). InnerVolumeSpecName "kube-api-access-j97pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:41.423828 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.423785 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609a0174-08f0-48a4-965d-02b0df4f4cd2-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:06:41.423828 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.423820 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j97pf\" (UniqueName: \"kubernetes.io/projected/609a0174-08f0-48a4-965d-02b0df4f4cd2-kube-api-access-j97pf\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:06:41.423828 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.423831 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609a0174-08f0-48a4-965d-02b0df4f4cd2-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:06:41.556786 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.556689 2573 generic.go:356] "Generic (PLEG): container finished" podID="609a0174-08f0-48a4-965d-02b0df4f4cd2" containerID="6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f" exitCode=0 Jul 06 20:06:41.556786 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.556729 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" Jul 06 20:06:41.557018 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.556732 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" event={"ID":"609a0174-08f0-48a4-965d-02b0df4f4cd2","Type":"ContainerDied","Data":"6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f"} Jul 06 20:06:41.557018 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.556853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn" event={"ID":"609a0174-08f0-48a4-965d-02b0df4f4cd2","Type":"ContainerDied","Data":"8bc5397acda0fd5b08410f9ba693539ba49221a875e7cb8234ad3b793796d3a5"} Jul 06 20:06:41.557018 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.556874 2573 scope.go:117] "RemoveContainer" containerID="6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f" Jul 06 20:06:41.566631 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.566609 2573 scope.go:117] "RemoveContainer" containerID="6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f" Jul 06 20:06:41.566958 ip-10-0-140-103 kubenswrapper[2573]: E0706 20:06:41.566938 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f\": container with ID starting with 6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f not found: ID does not exist" containerID="6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f" Jul 06 20:06:41.567032 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.566974 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f"} err="failed to get container status \"6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f\": rpc error: code = NotFound desc = could not find container \"6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f\": container with ID starting with 6d444d5a64b3b529839f84d610f0e9acaa61ef9a68c63cec01f16269de0b204f not found: ID does not exist" Jul 06 20:06:41.580731 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.580671 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn"] Jul 06 20:06:41.582679 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:41.582656 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/sub-test-model-c21a78-kserve-d55c88475-ktckn"] Jul 06 20:06:42.394373 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:42.394339 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="609a0174-08f0-48a4-965d-02b0df4f4cd2" path="/var/lib/kubelet/pods/609a0174-08f0-48a4-965d-02b0df4f4cd2/volumes" Jul 06 20:06:43.811410 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.811374 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl"] Jul 06 20:06:43.811809 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.811729 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" containerName="main" Jul 06 20:06:43.811809 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.811740 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" containerName="main" Jul 06 20:06:43.811809 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.811752 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609a0174-08f0-48a4-965d-02b0df4f4cd2" containerName="main" Jul 06 20:06:43.811809 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.811758 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="609a0174-08f0-48a4-965d-02b0df4f4cd2" containerName="main" Jul 06 20:06:43.811943 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.811815 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="609a0174-08f0-48a4-965d-02b0df4f4cd2" containerName="main" Jul 06 20:06:43.811943 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.811824 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f66d85-eaef-4a61-8af1-5e7ddd7b47e8" containerName="main" Jul 06 20:06:43.814709 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.814689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.817214 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.817189 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c21a78\"/\"kube-root-ca.crt\"" Jul 06 20:06:43.817888 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.817866 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c21a78\"/\"rate-test-model-c21a78-kserve-self-signed-certs\"" Jul 06 20:06:43.818000 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.817917 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c21a78\"/\"openshift-service-ca.crt\"" Jul 06 20:06:43.818000 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.817977 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c21a78\"/\"default-dockercfg-9nxz5\"" Jul 06 20:06:43.828867 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.828838 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl"] Jul 06 20:06:43.844677 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.844644 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-tmp-dir\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.844853 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.844686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a401-5315-4c80-a663-2fa29671f206-tls-certs\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.844853 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.844720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h99g\" (UniqueName: \"kubernetes.io/projected/a8e9a401-5315-4c80-a663-2fa29671f206-kube-api-access-6h99g\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.844853 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.844760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-model-cache\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.844853 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.844789 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-dshm\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.844853 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.844816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-home\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.945592 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.945552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h99g\" (UniqueName: \"kubernetes.io/projected/a8e9a401-5315-4c80-a663-2fa29671f206-kube-api-access-6h99g\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.945777 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.945608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-model-cache\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.945777 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.945647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-dshm\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.945777 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.945671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-home\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.945777 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.945703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-tmp-dir\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.945777 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.945734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a401-5315-4c80-a663-2fa29671f206-tls-certs\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.946101 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.946080 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-home\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.946153 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.946131 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-tmp-dir\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.946189 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.946174 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-model-cache\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.948222 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.948184 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-dshm\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.948346 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.948314 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a401-5315-4c80-a663-2fa29671f206-tls-certs\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:43.958560 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:43.958529 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h99g\" (UniqueName: \"kubernetes.io/projected/a8e9a401-5315-4c80-a663-2fa29671f206-kube-api-access-6h99g\") pod \"rate-test-model-c21a78-kserve-75b74d8d75-2bmtl\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:44.124661 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:44.124618 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:44.281252 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:44.280825 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl"] Jul 06 20:06:44.574890 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:44.574860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" event={"ID":"a8e9a401-5315-4c80-a663-2fa29671f206","Type":"ContainerStarted","Data":"96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb"} Jul 06 20:06:44.575055 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:44.574907 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" event={"ID":"a8e9a401-5315-4c80-a663-2fa29671f206","Type":"ContainerStarted","Data":"900c957f162f7a3094091f055d0cde4c35b3200eb2108120124b0e1b84fd2882"} Jul 06 20:06:44.612150 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:44.612095 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" podStartSLOduration=1.612076284 podStartE2EDuration="1.612076284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:44.605797638 +0000 UTC m=+2628.703025393" watchObservedRunningTime="2026-07-06 20:06:44.612076284 +0000 UTC m=+2628.709304038" Jul 06 20:06:54.125720 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:54.125662 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:54.126142 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:54.125743 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:54.138561 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:54.138528 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:54.642447 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:54.642413 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:06:56.593758 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:56.593725 2573 scope.go:117] "RemoveContainer" containerID="eadb4976f33b3b04fda604beb88570d48b42d09baf0bbc9a86fcb281dd05663b" Jul 06 20:06:56.602458 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:06:56.602434 2573 scope.go:117] "RemoveContainer" containerID="62d7ca54b60ea53f1d53a79e33dc552bfa0b79655b7e11166b5d57e88b2a6514" Jul 06 20:07:56.526625 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:07:56.526517 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 20:07:56.532153 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:07:56.532131 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 20:08:14.633626 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:14.633583 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl"] Jul 06 20:08:29.653806 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:29.653737 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" podUID="a8e9a401-5315-4c80-a663-2fa29671f206" containerName="main" containerID="cri-o://96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb" gracePeriod=45 Jul 06 20:08:29.898946 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:29.898921 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:08:30.026153 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026051 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-tmp-dir\") pod \"a8e9a401-5315-4c80-a663-2fa29671f206\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " Jul 06 20:08:30.026153 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026100 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-home\") pod \"a8e9a401-5315-4c80-a663-2fa29671f206\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " Jul 06 20:08:30.026153 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026156 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-dshm\") pod \"a8e9a401-5315-4c80-a663-2fa29671f206\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " Jul 06 20:08:30.026480 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026177 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a401-5315-4c80-a663-2fa29671f206-tls-certs\") pod \"a8e9a401-5315-4c80-a663-2fa29671f206\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " Jul 06 20:08:30.026480 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026198 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h99g\" (UniqueName: \"kubernetes.io/projected/a8e9a401-5315-4c80-a663-2fa29671f206-kube-api-access-6h99g\") pod \"a8e9a401-5315-4c80-a663-2fa29671f206\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " Jul 06 20:08:30.026480 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026258 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-model-cache\") pod \"a8e9a401-5315-4c80-a663-2fa29671f206\" (UID: \"a8e9a401-5315-4c80-a663-2fa29671f206\") " Jul 06 20:08:30.026480 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026448 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8e9a401-5315-4c80-a663-2fa29671f206" (UID: "a8e9a401-5315-4c80-a663-2fa29671f206"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:30.026664 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026558 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-home" (OuterVolumeSpecName: "home") pod "a8e9a401-5315-4c80-a663-2fa29671f206" (UID: "a8e9a401-5315-4c80-a663-2fa29671f206"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:30.026664 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026605 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-tmp-dir\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:08:30.026664 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.026627 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8e9a401-5315-4c80-a663-2fa29671f206" (UID: "a8e9a401-5315-4c80-a663-2fa29671f206"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:30.028557 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.028523 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e9a401-5315-4c80-a663-2fa29671f206-kube-api-access-6h99g" (OuterVolumeSpecName: "kube-api-access-6h99g") pod "a8e9a401-5315-4c80-a663-2fa29671f206" (UID: "a8e9a401-5315-4c80-a663-2fa29671f206"). InnerVolumeSpecName "kube-api-access-6h99g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:08:30.028805 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.028785 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-dshm" (OuterVolumeSpecName: "dshm") pod "a8e9a401-5315-4c80-a663-2fa29671f206" (UID: "a8e9a401-5315-4c80-a663-2fa29671f206"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:30.028900 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.028803 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e9a401-5315-4c80-a663-2fa29671f206-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8e9a401-5315-4c80-a663-2fa29671f206" (UID: "a8e9a401-5315-4c80-a663-2fa29671f206"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:08:30.127072 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.127031 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-dshm\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:08:30.127072 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.127062 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e9a401-5315-4c80-a663-2fa29671f206-tls-certs\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:08:30.127072 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.127076 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h99g\" (UniqueName: \"kubernetes.io/projected/a8e9a401-5315-4c80-a663-2fa29671f206-kube-api-access-6h99g\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:08:30.127072 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.127086 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-model-cache\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:08:30.127391 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.127107 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e9a401-5315-4c80-a663-2fa29671f206-home\") on node \"ip-10-0-140-103.ec2.internal\" DevicePath \"\"" Jul 06 20:08:30.139999 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.139966 2573 generic.go:356] "Generic (PLEG): container finished" podID="a8e9a401-5315-4c80-a663-2fa29671f206" containerID="96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb" exitCode=0 Jul 06 20:08:30.140182 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.140019 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" Jul 06 20:08:30.140182 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.140029 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" event={"ID":"a8e9a401-5315-4c80-a663-2fa29671f206","Type":"ContainerDied","Data":"96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb"} Jul 06 20:08:30.140182 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.140060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl" event={"ID":"a8e9a401-5315-4c80-a663-2fa29671f206","Type":"ContainerDied","Data":"900c957f162f7a3094091f055d0cde4c35b3200eb2108120124b0e1b84fd2882"} Jul 06 20:08:30.140182 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.140078 2573 scope.go:117] "RemoveContainer" containerID="96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb" Jul 06 20:08:30.150056 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.150038 2573 scope.go:117] "RemoveContainer" containerID="96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb" Jul 06 20:08:30.150357 ip-10-0-140-103 kubenswrapper[2573]: E0706 20:08:30.150336 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb\": container with ID starting with 96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb not found: ID does not exist" containerID="96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb" Jul 06 20:08:30.150433 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.150368 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb"} err="failed to get container status \"96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb\": rpc error: code = NotFound desc = could not find container \"96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb\": container with ID starting with 96c67fa575548e0119ba872ac51f6460c96536011f0e53baa7112b92525c9ebb not found: ID does not exist" Jul 06 20:08:30.163818 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.163778 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl"] Jul 06 20:08:30.173176 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.173140 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c21a78/rate-test-model-c21a78-kserve-75b74d8d75-2bmtl"] Jul 06 20:08:30.394488 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:08:30.394457 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e9a401-5315-4c80-a663-2fa29671f206" path="/var/lib/kubelet/pods/a8e9a401-5315-4c80-a663-2fa29671f206/volumes" Jul 06 20:09:15.156787 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:15.156758 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722800-5ggls_c3659e28-327f-48b5-9732-4bff9d8464c3/cleanup/0.log" Jul 06 20:09:15.793398 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:15.793370 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7865995f8c-q6vmg_87de029c-e31a-42a6-afc3-476eb4834a9c/copy-manifests/0.log" Jul 06 20:09:15.806708 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:15.806662 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7865995f8c-q6vmg_87de029c-e31a-42a6-afc3-476eb4834a9c/manager/0.log" Jul 06 20:09:16.945850 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:16.945821 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc_0470f7d5-5f99-4019-9eea-336475adb661/util/0.log" Jul 06 20:09:16.953712 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:16.953682 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc_0470f7d5-5f99-4019-9eea-336475adb661/pull/0.log" Jul 06 20:09:16.960136 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:16.960113 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc_0470f7d5-5f99-4019-9eea-336475adb661/extract/0.log" Jul 06 20:09:17.081365 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.081332 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9_1523ea4a-cced-42ce-b7d2-532ec4baf1f5/util/0.log" Jul 06 20:09:17.093221 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.093179 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9_1523ea4a-cced-42ce-b7d2-532ec4baf1f5/pull/0.log" Jul 06 20:09:17.108458 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.108431 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9_1523ea4a-cced-42ce-b7d2-532ec4baf1f5/extract/0.log" Jul 06 20:09:17.232182 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.232069 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 20:09:17.267781 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.267753 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/util/0.log" Jul 06 20:09:17.273900 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.273875 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/pull/0.log" Jul 06 20:09:17.410917 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.410886 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq_bbaeecaa-91be-4793-8228-e685b108c266/util/0.log" Jul 06 20:09:17.420553 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.420528 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq_bbaeecaa-91be-4793-8228-e685b108c266/pull/0.log" Jul 06 20:09:17.432545 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.432520 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq_bbaeecaa-91be-4793-8228-e685b108c266/extract/0.log" Jul 06 20:09:17.549791 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.549712 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74_b05a45cd-a8d4-4e53-9a0e-d373374a70e0/util/0.log" Jul 06 20:09:17.561839 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.561811 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74_b05a45cd-a8d4-4e53-9a0e-d373374a70e0/pull/0.log" Jul 06 20:09:17.575423 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:17.575396 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74_b05a45cd-a8d4-4e53-9a0e-d373374a70e0/extract/0.log" Jul 06 20:09:18.350400 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:18.350273 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-x97z2_94247084-7c3d-4b76-a4b9-72d708348353/manager/0.log" Jul 06 20:09:19.050220 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:19.050175 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-fplgk_6686c259-eb37-4cba-b4b7-8fb2a04b4b36/discovery/0.log" Jul 06 20:09:19.513948 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:19.513912 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-g25zj_f046b51b-34bb-4e1b-aba0-88431a3f1422/payload-pre-processing/0.log" Jul 06 20:09:20.090912 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:20.090881 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-gjbl6_5c1717cd-5944-4288-a859-6dca347a45b8/main/0.log" Jul 06 20:09:20.566322 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:20.566246 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-9v5px_39f605cb-86b3-4cf2-be88-a7ef37924ec7/main/0.log" Jul 06 20:09:24.609964 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.609923 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tfs5f/must-gather-4sdb7"] Jul 06 20:09:24.610503 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.610345 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e9a401-5315-4c80-a663-2fa29671f206" containerName="main" Jul 06 20:09:24.610503 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.610363 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e9a401-5315-4c80-a663-2fa29671f206" containerName="main" Jul 06 20:09:24.610503 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.610478 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e9a401-5315-4c80-a663-2fa29671f206" containerName="main" Jul 06 20:09:24.613542 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.613517 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfs5f/must-gather-4sdb7" Jul 06 20:09:24.619015 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.618984 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfs5f\"/\"openshift-service-ca.crt\"" Jul 06 20:09:24.619242 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.618984 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tfs5f\"/\"default-dockercfg-2txjc\"" Jul 06 20:09:24.619818 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.619799 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfs5f\"/\"kube-root-ca.crt\"" Jul 06 20:09:24.622329 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.622303 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfs5f/must-gather-4sdb7"] Jul 06 20:09:24.728535 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.728498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/140982ff-b4f9-483e-80c0-ce2571b4dd01-must-gather-output\") pod \"must-gather-4sdb7\" (UID: \"140982ff-b4f9-483e-80c0-ce2571b4dd01\") " pod="openshift-must-gather-tfs5f/must-gather-4sdb7" Jul 06 20:09:24.728716 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.728630 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9m4q\" (UniqueName: \"kubernetes.io/projected/140982ff-b4f9-483e-80c0-ce2571b4dd01-kube-api-access-r9m4q\") pod \"must-gather-4sdb7\" (UID: \"140982ff-b4f9-483e-80c0-ce2571b4dd01\") " pod="openshift-must-gather-tfs5f/must-gather-4sdb7" Jul 06 20:09:24.829324 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.829282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9m4q\" (UniqueName: \"kubernetes.io/projected/140982ff-b4f9-483e-80c0-ce2571b4dd01-kube-api-access-r9m4q\") pod \"must-gather-4sdb7\" (UID: \"140982ff-b4f9-483e-80c0-ce2571b4dd01\") " pod="openshift-must-gather-tfs5f/must-gather-4sdb7" Jul 06 20:09:24.829498 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.829346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/140982ff-b4f9-483e-80c0-ce2571b4dd01-must-gather-output\") pod \"must-gather-4sdb7\" (UID: \"140982ff-b4f9-483e-80c0-ce2571b4dd01\") " pod="openshift-must-gather-tfs5f/must-gather-4sdb7" Jul 06 20:09:24.829661 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.829644 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/140982ff-b4f9-483e-80c0-ce2571b4dd01-must-gather-output\") pod \"must-gather-4sdb7\" (UID: \"140982ff-b4f9-483e-80c0-ce2571b4dd01\") " pod="openshift-must-gather-tfs5f/must-gather-4sdb7" Jul 06 20:09:24.874099 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.874008 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9m4q\" (UniqueName: \"kubernetes.io/projected/140982ff-b4f9-483e-80c0-ce2571b4dd01-kube-api-access-r9m4q\") pod \"must-gather-4sdb7\" (UID: \"140982ff-b4f9-483e-80c0-ce2571b4dd01\") " pod="openshift-must-gather-tfs5f/must-gather-4sdb7" Jul 06 20:09:24.924574 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:24.924524 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfs5f/must-gather-4sdb7" Jul 06 20:09:25.080511 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:25.080486 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfs5f/must-gather-4sdb7"] Jul 06 20:09:25.082394 ip-10-0-140-103 kubenswrapper[2573]: W0706 20:09:25.082361 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod140982ff_b4f9_483e_80c0_ce2571b4dd01.slice/crio-4ded69edb100beb8a87ca887209efc6a3fd5d3029a0c020a177f1358c009d50a WatchSource:0}: Error finding container 4ded69edb100beb8a87ca887209efc6a3fd5d3029a0c020a177f1358c009d50a: Status 404 returned error can't find the container with id 4ded69edb100beb8a87ca887209efc6a3fd5d3029a0c020a177f1358c009d50a Jul 06 20:09:25.084372 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:25.084352 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:09:25.428961 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:25.428923 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfs5f/must-gather-4sdb7" event={"ID":"140982ff-b4f9-483e-80c0-ce2571b4dd01","Type":"ContainerStarted","Data":"4ded69edb100beb8a87ca887209efc6a3fd5d3029a0c020a177f1358c009d50a"} Jul 06 20:09:26.436445 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:26.436405 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfs5f/must-gather-4sdb7" event={"ID":"140982ff-b4f9-483e-80c0-ce2571b4dd01","Type":"ContainerStarted","Data":"06793b9d11ec12e4d6cd32ae954d0f29330a1a50a55746a46ab9043fb157d357"} Jul 06 20:09:26.436924 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:26.436463 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfs5f/must-gather-4sdb7" event={"ID":"140982ff-b4f9-483e-80c0-ce2571b4dd01","Type":"ContainerStarted","Data":"99fa93e0f2b5b52f55335ccd8a3e6ee0e739effbbb99884cd67b1c0a084d05d6"} Jul 06 20:09:26.469613 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:26.469540 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tfs5f/must-gather-4sdb7" podStartSLOduration=1.73646425 podStartE2EDuration="2.469514437s" totalImagesPullingTime="733.050187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:09:26.466959191 +0000 UTC m=+2790.564186948" watchObservedRunningTime="2026-07-06 20:09:26.469514437 +0000 UTC m=+2790.566742195" Jul 06 20:09:27.998404 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:27.998365 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bsvw4_6f20bf0b-a1f4-43f3-800d-a892bff4c7fe/global-pull-secret-syncer/0.log" Jul 06 20:09:28.191644 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:28.191609 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kbgkq_f79f1647-3ed9-4e04-8589-2b547e769cc9/konnectivity-agent/0.log" Jul 06 20:09:28.298005 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:28.297925 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-103.ec2.internal_ce19eea5e92a6ee78e17d806a018297b/haproxy/0.log" Jul 06 20:09:32.660219 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.660172 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc_0470f7d5-5f99-4019-9eea-336475adb661/extract/0.log" Jul 06 20:09:32.701032 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.701001 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc_0470f7d5-5f99-4019-9eea-336475adb661/util/0.log" Jul 06 20:09:32.735389 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.735362 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gm8sc_0470f7d5-5f99-4019-9eea-336475adb661/pull/0.log" Jul 06 20:09:32.780767 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.780726 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9_1523ea4a-cced-42ce-b7d2-532ec4baf1f5/extract/0.log" Jul 06 20:09:32.817933 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.817907 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9_1523ea4a-cced-42ce-b7d2-532ec4baf1f5/util/0.log" Jul 06 20:09:32.848636 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.848585 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wbkh9_1523ea4a-cced-42ce-b7d2-532ec4baf1f5/pull/0.log" Jul 06 20:09:32.888093 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.888066 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/extract/0.log" Jul 06 20:09:32.923514 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.923305 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/util/0.log" Jul 06 20:09:32.961576 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.961548 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73765ns_7fbeec71-aed1-4625-9030-2431a05a7145/pull/0.log" Jul 06 20:09:32.996182 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:32.996144 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq_bbaeecaa-91be-4793-8228-e685b108c266/extract/0.log" Jul 06 20:09:33.025031 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:33.024951 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq_bbaeecaa-91be-4793-8228-e685b108c266/util/0.log" Jul 06 20:09:33.061007 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:33.060965 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m8pnq_bbaeecaa-91be-4793-8228-e685b108c266/pull/0.log" Jul 06 20:09:33.092032 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:33.092007 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74_b05a45cd-a8d4-4e53-9a0e-d373374a70e0/extract/0.log" Jul 06 20:09:33.131183 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:33.131154 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74_b05a45cd-a8d4-4e53-9a0e-d373374a70e0/util/0.log" Jul 06 20:09:33.162594 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:33.162555 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1czx74_b05a45cd-a8d4-4e53-9a0e-d373374a70e0/pull/0.log" Jul 06 20:09:33.862074 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:33.862038 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-x97z2_94247084-7c3d-4b76-a4b9-72d708348353/manager/0.log" Jul 06 20:09:36.455159 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:36.455122 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7c4684648-cmjsv_fff59c3d-13b5-42cb-b3ea-6efcd180b04c/metrics-server/0.log" Jul 06 20:09:36.548189 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:36.548153 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-842r7_2efca560-1b2f-45e1-93e7-75a60e94d465/node-exporter/0.log" Jul 06 20:09:36.584056 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:36.584027 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-842r7_2efca560-1b2f-45e1-93e7-75a60e94d465/kube-rbac-proxy/0.log" Jul 06 20:09:36.649291 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:36.649263 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-842r7_2efca560-1b2f-45e1-93e7-75a60e94d465/init-textfile/0.log" Jul 06 20:09:36.932121 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:36.932062 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-8ngjj_06c1378b-4fb3-4b87-ac27-cd84f0a1793c/kube-rbac-proxy-main/0.log" Jul 06 20:09:36.972117 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:36.972076 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-8ngjj_06c1378b-4fb3-4b87-ac27-cd84f0a1793c/kube-rbac-proxy-self/0.log" Jul 06 20:09:37.001708 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.001677 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-8ngjj_06c1378b-4fb3-4b87-ac27-cd84f0a1793c/openshift-state-metrics/0.log" Jul 06 20:09:37.602300 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.602267 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f86dccc9f-zwmvg_5cf97e2c-9b20-42b0-ac71-79b006de51dc/telemeter-client/0.log" Jul 06 20:09:37.668061 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.668015 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f86dccc9f-zwmvg_5cf97e2c-9b20-42b0-ac71-79b006de51dc/reload/0.log" Jul 06 20:09:37.715649 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.715605 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f86dccc9f-zwmvg_5cf97e2c-9b20-42b0-ac71-79b006de51dc/kube-rbac-proxy/0.log" Jul 06 20:09:37.781284 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.781234 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d689b697b-qd29t_55065c34-242e-474c-86f2-dbaf200768f1/thanos-query/0.log" Jul 06 20:09:37.823417 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.823354 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d689b697b-qd29t_55065c34-242e-474c-86f2-dbaf200768f1/kube-rbac-proxy-web/0.log" Jul 06 20:09:37.872116 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.872000 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d689b697b-qd29t_55065c34-242e-474c-86f2-dbaf200768f1/kube-rbac-proxy/0.log" Jul 06 20:09:37.904828 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.904781 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5"] Jul 06 20:09:37.910106 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.910075 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:37.949045 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.949006 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5"] Jul 06 20:09:37.966943 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.966904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmwq4\" (UniqueName: \"kubernetes.io/projected/b08ece62-47de-4187-a19e-2c7d9cbf768c-kube-api-access-kmwq4\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:37.967223 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.967188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-proc\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:37.967397 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.967382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-podres\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:37.967515 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.967501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-sys\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:37.967643 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.967630 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-lib-modules\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:37.974869 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:37.974828 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d689b697b-qd29t_55065c34-242e-474c-86f2-dbaf200768f1/prom-label-proxy/0.log" Jul 06 20:09:38.031222 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.031187 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d689b697b-qd29t_55065c34-242e-474c-86f2-dbaf200768f1/kube-rbac-proxy-rules/0.log" Jul 06 20:09:38.068116 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.068070 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-sys\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.068304 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.068133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-lib-modules\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.068304 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.068183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmwq4\" (UniqueName: \"kubernetes.io/projected/b08ece62-47de-4187-a19e-2c7d9cbf768c-kube-api-access-kmwq4\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.068304 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.068244 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-sys\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.068304 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.068283 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-lib-modules\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.068470 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.068310 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-proc\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.068470 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.068387 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-podres\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.068470 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.068402 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-proc\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.068582 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.068491 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b08ece62-47de-4187-a19e-2c7d9cbf768c-podres\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.074577 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.074550 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d689b697b-qd29t_55065c34-242e-474c-86f2-dbaf200768f1/kube-rbac-proxy-metrics/0.log" Jul 06 20:09:38.081736 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.081704 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmwq4\" (UniqueName: \"kubernetes.io/projected/b08ece62-47de-4187-a19e-2c7d9cbf768c-kube-api-access-kmwq4\") pod \"perf-node-gather-daemonset-qpfc5\" (UID: \"b08ece62-47de-4187-a19e-2c7d9cbf768c\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.223862 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.223773 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.423935 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.423906 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5"] Jul 06 20:09:38.426628 ip-10-0-140-103 kubenswrapper[2573]: W0706 20:09:38.426592 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb08ece62_47de_4187_a19e_2c7d9cbf768c.slice/crio-58308cc54f3a31d61d81f9ce6b6644bc12706d51f23adf692673e05a508708fb WatchSource:0}: Error finding container 58308cc54f3a31d61d81f9ce6b6644bc12706d51f23adf692673e05a508708fb: Status 404 returned error can't find the container with id 58308cc54f3a31d61d81f9ce6b6644bc12706d51f23adf692673e05a508708fb Jul 06 20:09:38.556757 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.556713 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" event={"ID":"b08ece62-47de-4187-a19e-2c7d9cbf768c","Type":"ContainerStarted","Data":"42c06e4ac04672a7deb38f578f37c683c43c6e6580f8da9d2584e11b3e1e9cf3"} Jul 06 20:09:38.556930 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.556784 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" event={"ID":"b08ece62-47de-4187-a19e-2c7d9cbf768c","Type":"ContainerStarted","Data":"58308cc54f3a31d61d81f9ce6b6644bc12706d51f23adf692673e05a508708fb"} Jul 06 20:09:38.556930 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.556825 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:38.592487 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:38.592404 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" podStartSLOduration=1.592380919 podStartE2EDuration="1.592380919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:09:38.591973594 +0000 UTC m=+2802.689201350" watchObservedRunningTime="2026-07-06 20:09:38.592380919 +0000 UTC m=+2802.689608675" Jul 06 20:09:40.495780 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:40.495753 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56cf7cd8-vc72j_e0e3c94a-f1d6-4758-8ea1-e093d82a1459/console/0.log" Jul 06 20:09:42.577580 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:42.577546 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sx4t5_79911034-6ae1-42d7-9b8a-4f370e0b3099/dns/0.log" Jul 06 20:09:42.605651 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:42.605604 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sx4t5_79911034-6ae1-42d7-9b8a-4f370e0b3099/kube-rbac-proxy/0.log" Jul 06 20:09:42.632511 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:42.632477 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gdpq6_e0eeb8e3-8400-49e8-89cb-3977de6b1b6d/dns-node-resolver/0.log" Jul 06 20:09:43.366196 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:43.366166 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f59cc6f6d-bnxlq_65362076-a446-4576-ad83-6a994035a453/registry/0.log" Jul 06 20:09:43.512880 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:43.512853 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qjr9z_ce473c59-a118-4568-99b6-c67a261353e6/node-ca/0.log" Jul 06 20:09:44.576551 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:44.576502 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-qpfc5" Jul 06 20:09:44.846152 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:44.846067 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-fplgk_6686c259-eb37-4cba-b4b7-8fb2a04b4b36/discovery/0.log" Jul 06 20:09:45.046716 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:45.046686 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-g25zj_f046b51b-34bb-4e1b-aba0-88431a3f1422/payload-pre-processing/0.log" Jul 06 20:09:46.316872 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:46.316838 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v85qf_8bd697ac-eb1a-4cda-a72a-6024967509f4/serve-healthcheck-canary/0.log" Jul 06 20:09:47.736315 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:47.736286 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czqtk_c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4/kube-rbac-proxy/0.log" Jul 06 20:09:47.776041 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:47.776008 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czqtk_c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4/exporter/0.log" Jul 06 20:09:47.821339 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:47.821306 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czqtk_c75ebb1d-b187-4fb5-b4bf-5cb7cbe217f4/extractor/0.log" Jul 06 20:09:49.700428 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:49.700391 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-fplgk_6686c259-eb37-4cba-b4b7-8fb2a04b4b36/discovery/0.log" Jul 06 20:09:49.827750 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:49.827719 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-g25zj_f046b51b-34bb-4e1b-aba0-88431a3f1422/payload-pre-processing/0.log" Jul 06 20:09:50.543806 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:50.543743 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722800-5ggls_c3659e28-327f-48b5-9732-4bff9d8464c3/cleanup/0.log" Jul 06 20:09:50.969905 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:50.969870 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7865995f8c-q6vmg_87de029c-e31a-42a6-afc3-476eb4834a9c/manager/0.log" Jul 06 20:09:50.992015 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:50.991992 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7865995f8c-q6vmg_87de029c-e31a-42a6-afc3-476eb4834a9c/copy-manifests/0.log" Jul 06 20:09:51.237296 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:51.236479 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-fplgk_6686c259-eb37-4cba-b4b7-8fb2a04b4b36/discovery/0.log" Jul 06 20:09:51.398052 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:51.398022 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-g25zj_f046b51b-34bb-4e1b-aba0-88431a3f1422/payload-pre-processing/0.log" Jul 06 20:09:52.581438 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:52.581402 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54d6cd8878-9x5fq_e17f75ab-ddd9-4fe5-8a4b-b8f969a45874/manager/2.log" Jul 06 20:09:52.581905 ip-10-0-140-103 kubenswrapper[2573]: I0706 20:09:52.581842 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54d6cd8878-9x5fq_e17f75ab-ddd9-4fe5-8a4b-b8f969a45874/manager/1.log"