Jul 21 08:18:47.809678 ip-10-0-137-36 systemd[1]: Starting Kubernetes Kubelet... Jul 21 08:18:48.159424 ip-10-0-137-36 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 08:18:48.159424 ip-10-0-137-36 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 08:18:48.159424 ip-10-0-137-36 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 08:18:48.159424 ip-10-0-137-36 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 08:18:48.159424 ip-10-0-137-36 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 08:18:48.160406 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.160269 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 08:18:48.164876 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164857 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:18:48.164876 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164876 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164883 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164887 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164892 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164897 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164901 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164906 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164910 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164913 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164917 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164921 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164926 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164931 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164936 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164941 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164945 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164950 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164954 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164958 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:18:48.165027 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164962 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164971 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164975 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164979 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164984 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164988 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164992 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.164996 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165001 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165005 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165009 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165013 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165017 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165021 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165025 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165029 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165034 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165038 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165042 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165046 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:18:48.165822 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165050 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165054 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165058 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165062 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165067 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165071 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165075 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165079 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165085 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165089 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165093 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165098 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165102 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165107 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165112 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165116 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165121 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165124 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165129 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165135 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:18:48.166656 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165139 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165143 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165149 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165154 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165157 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165162 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165166 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165171 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165175 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165179 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165184 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165188 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165192 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165196 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165200 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165204 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165208 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165214 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165219 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:18:48.167517 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165224 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165229 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165233 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165237 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165242 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165246 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165249 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165254 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165258 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165263 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165268 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165272 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165275 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165280 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165284 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165288 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165292 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165297 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165302 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.165306 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:18:48.168309 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166480 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166491 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166495 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166500 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166504 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166509 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166513 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166517 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166522 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166526 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166530 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166534 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166538 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166542 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166546 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166550 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166554 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166559 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166563 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166567 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:18:48.169157 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166571 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166576 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166581 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166585 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166589 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166593 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166597 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166602 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166606 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166611 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166615 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166620 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166624 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166631 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166637 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166642 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166646 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166652 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166658 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166662 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:18:48.169985 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166666 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166670 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166674 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166678 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166682 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166686 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166693 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166698 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166704 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166708 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166713 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166718 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166722 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166726 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166732 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166744 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166749 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166753 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166757 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166764 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:18:48.170525 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166769 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166773 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166777 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166781 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166785 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166789 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166794 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166798 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166802 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166806 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166810 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166814 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166819 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166823 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166827 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166831 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166835 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166839 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166844 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166848 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:18:48.171031 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166854 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166858 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166862 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166867 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166872 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166880 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166885 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166890 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166895 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166899 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166903 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166907 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166912 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166916 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166921 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166925 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166929 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166934 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.166938 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:18:48.171538 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167581 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167590 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167596 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167601 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167606 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167610 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167615 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167619 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167623 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167627 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167631 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167636 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167641 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167645 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167649 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167654 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167659 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167663 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167670 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167674 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:18:48.171997 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167680 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167687 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167693 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167699 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167703 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167709 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167716 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167721 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167725 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167730 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167735 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167739 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167743 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167748 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167752 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167756 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167762 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167766 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167770 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:18:48.172499 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167775 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167779 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167783 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167787 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167791 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167795 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167799 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167804 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167810 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167815 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167819 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167824 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167828 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167832 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167837 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167841 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167846 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167850 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167854 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167859 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:18:48.172950 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167863 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167867 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167871 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167875 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167880 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167884 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167888 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167892 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167896 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167900 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167904 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167909 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167913 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167918 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167922 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167926 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167930 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167936 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167942 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:18:48.173458 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167947 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167953 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167957 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167962 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167968 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167973 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167977 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167982 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167987 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167991 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.167997 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168001 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168006 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168010 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168015 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168019 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168023 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168027 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168031 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:18:48.173913 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168036 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.168040 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168136 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168147 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168157 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168164 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168171 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168176 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168183 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168189 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168194 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168199 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168205 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168210 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168217 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168222 2563 flags.go:64] FLAG: --cgroup-root="" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168227 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168233 2563 flags.go:64] FLAG: --client-ca-file="" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168238 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168243 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168249 2563 flags.go:64] FLAG: --cluster-domain="" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168254 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168259 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168265 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 08:18:48.174444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168270 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168276 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168281 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168286 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168291 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168296 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168300 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168306 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168311 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168315 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168321 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168325 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168330 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168335 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168340 2563 flags.go:64] FLAG: --enable-server="true" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168344 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168352 2563 flags.go:64] FLAG: --event-burst="100" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168357 2563 flags.go:64] FLAG: --event-qps="50" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168362 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168367 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168371 2563 flags.go:64] FLAG: --eviction-hard="" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168377 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168403 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168408 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168413 2563 flags.go:64] FLAG: --eviction-soft="" Jul 21 08:18:48.175040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168419 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168423 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168428 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168432 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168437 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168441 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168446 2563 flags.go:64] FLAG: --feature-gates="" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168452 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168457 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168462 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168467 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168472 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168477 2563 flags.go:64] FLAG: --help="false" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168482 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168487 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168491 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168496 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168501 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168507 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168512 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168516 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168521 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168525 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 08:18:48.175648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168530 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168535 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168540 2563 flags.go:64] FLAG: --kube-reserved="" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168545 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168549 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168553 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168560 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168564 2563 flags.go:64] FLAG: --lock-file="" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168568 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168574 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168579 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168587 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168592 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168597 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168601 2563 flags.go:64] FLAG: --logging-format="text" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168606 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168612 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168618 2563 flags.go:64] FLAG: --manifest-url="" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168622 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168630 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168635 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168641 2563 flags.go:64] FLAG: --max-pods="110" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168646 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168650 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168655 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 08:18:48.176227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168660 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168665 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168669 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168674 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168685 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168689 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168694 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168699 2563 flags.go:64] FLAG: --pod-cidr="" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168703 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168712 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168716 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168721 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168726 2563 flags.go:64] FLAG: --port="10250" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168732 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168737 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ac72c58774f458eb" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168742 2563 flags.go:64] FLAG: --qos-reserved="" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168747 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168751 2563 flags.go:64] FLAG: --register-node="true" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168756 2563 flags.go:64] FLAG: --register-with-taints="" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168762 2563 flags.go:64] FLAG: --registry-burst="10" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168766 2563 flags.go:64] FLAG: --registry-qps="5" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168771 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168776 2563 flags.go:64] FLAG: --reserved-memory="" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168782 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168787 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 08:18:48.176945 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168792 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168797 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168802 2563 flags.go:64] FLAG: --runonce="false" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168806 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168811 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168816 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168820 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168825 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168829 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168834 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168839 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168843 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168848 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168852 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168857 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168861 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168867 2563 flags.go:64] FLAG: --system-cgroups="" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168871 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168879 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168884 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168890 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168897 2563 flags.go:64] FLAG: --tls-min-version="" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168901 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168906 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168910 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 08:18:48.177624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168915 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168921 2563 flags.go:64] FLAG: --v="2" Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168927 2563 flags.go:64] FLAG: --version="false" Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168933 2563 flags.go:64] FLAG: --vmodule="" Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168939 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.168944 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170284 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170298 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170302 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170305 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170311 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170315 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170319 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170321 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170324 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170327 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170330 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170333 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170335 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170338 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:18:48.178236 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170341 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170344 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170347 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170350 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170353 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170356 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170359 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170361 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170364 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170367 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170370 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170373 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170375 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170378 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170394 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170397 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170400 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170403 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170405 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:18:48.178755 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170408 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170411 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170414 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170416 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170419 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170422 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170424 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170427 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170429 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170432 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170435 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170437 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170439 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170442 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170445 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170447 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170450 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170453 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170456 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170459 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:18:48.179213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170461 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170464 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170467 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170471 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170474 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170477 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170480 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170483 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170486 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170488 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170492 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170494 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170498 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170501 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170504 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170506 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170509 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170512 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170514 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170517 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:18:48.179723 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170519 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170522 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170527 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170531 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170534 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170536 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170539 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170542 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170544 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170547 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170550 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170553 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170555 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170558 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170561 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170563 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170565 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170568 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170570 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170573 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:18:48.180195 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170576 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170578 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170582 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170584 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170587 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.170590 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.171154 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.177921 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.177936 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.177955 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.177963 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178017 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178022 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178025 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:18:48.180685 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178027 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178030 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178033 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178035 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178038 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178041 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178043 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178046 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178049 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178052 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178054 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178057 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178059 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178062 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178064 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178067 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178069 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178072 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178075 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178077 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:18:48.181043 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178080 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178083 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178085 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178088 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178090 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178093 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178096 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178100 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178104 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178107 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178110 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178113 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178115 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178118 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178120 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178123 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178125 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178128 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178130 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:18:48.181651 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178133 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178135 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178139 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178141 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178143 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178146 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178149 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178152 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178156 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178159 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178162 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178165 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178168 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178171 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178173 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178176 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178179 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178182 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178184 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:18:48.182126 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178188 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178190 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178193 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178196 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178198 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178201 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178204 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178206 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178209 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178211 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178214 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178216 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178219 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178221 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178224 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178227 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178230 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178232 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178235 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178237 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:18:48.182603 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178240 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178246 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178250 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178253 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178256 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178260 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178263 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178266 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178268 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178271 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178274 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178277 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178280 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178283 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178286 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178289 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178292 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:18:48.183091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178294 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.178299 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178427 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178433 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178436 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178439 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178442 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178444 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178447 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178450 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178452 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178455 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178458 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178461 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178463 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:18:48.183539 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178466 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178468 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178471 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178473 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178476 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178478 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178480 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178483 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178485 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178488 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178490 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178493 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178496 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178499 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178502 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178505 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178507 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178510 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178512 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178515 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:18:48.183922 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178518 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178520 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178523 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178525 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178528 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178530 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178533 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178535 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178537 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178540 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178543 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178545 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178548 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178550 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178553 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178555 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178558 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178560 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178562 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:18:48.184424 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178565 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178568 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178570 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178573 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178575 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178578 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178581 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178583 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178585 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178588 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178591 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178593 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178596 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178598 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178600 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178603 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178605 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178608 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178610 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178613 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:18:48.185201 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178615 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178618 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178620 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178622 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178625 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178629 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178632 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178635 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178638 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178640 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178643 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178647 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178649 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178652 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178655 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178657 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178660 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178663 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178666 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:18:48.186024 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178669 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178671 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178674 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178676 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178679 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178681 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178685 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:48.178688 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.178693 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.178805 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.181166 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.182488 2563 server.go:1019] "Starting client certificate rotation" Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.182598 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 08:18:48.186829 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.183363 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 08:18:48.187236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.185082 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 08:18:48.187236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.187005 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 08:18:48.199546 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.199531 2563 log.go:25] "Validated CRI v1 runtime API" Jul 21 08:18:48.202939 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.202920 2563 log.go:25] "Validated CRI v1 image API" Jul 21 08:18:48.204174 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.204154 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 08:18:48.206616 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.206594 2563 fs.go:135] Filesystem UUIDs: map[5726e528-47b2-4ab2-a3bb-5fc5ba622356:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 92d29377-fec5-4276-99b3-87da1d49f4a0:/dev/nvme0n1p3] Jul 21 08:18:48.206675 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.206616 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 08:18:48.211937 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.211918 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 08:18:48.212513 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.212406 2563 manager.go:216] Machine: {Timestamp:2026-07-21 08:18:48.211038935 +0000 UTC m=+0.290077259 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099788 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2376f3af22370d5ea74b2da6b62147 SystemUUID:ec2376f3-af22-370d-5ea7-4b2da6b62147 BootID:43a12a9f-a629-43e0-a796-b9e078988c02 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:a1:76:fb:13 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:a1:76:fb:13 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:fc:2d:1c:4d:37 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 08:18:48.212513 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.212512 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 08:18:48.212606 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.212593 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 08:18:48.215618 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.215594 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 08:18:48.215752 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.215621 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-36.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 08:18:48.216879 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.216869 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 08:18:48.216915 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.216881 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 08:18:48.216915 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.216889 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 08:18:48.217493 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.217483 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 08:18:48.218937 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.218926 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 08:18:48.221946 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.221935 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 21 08:18:48.222067 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.222057 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 08:18:48.222103 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.222088 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 21 08:18:48.222103 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.222098 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 08:18:48.222155 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.222109 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 08:18:48.222155 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.222120 2563 kubelet.go:398] "Adding apiserver pod source" Jul 21 08:18:48.222155 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.222129 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 08:18:48.222580 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.222568 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 08:18:48.222619 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.222588 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 08:18:48.224452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.224432 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 08:18:48.226396 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.226367 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 08:18:48.226450 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.226421 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 08:18:48.227508 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227498 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 08:18:48.227558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227514 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 08:18:48.227558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227519 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 08:18:48.227558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227524 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 08:18:48.227558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227533 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 08:18:48.227558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227539 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 08:18:48.227558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227548 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 08:18:48.227558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227553 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 08:18:48.227558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227559 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 08:18:48.227769 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227564 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 08:18:48.227769 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227581 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 08:18:48.227769 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.227587 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 08:18:48.228227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.228217 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 08:18:48.228259 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.228227 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 08:18:48.231673 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.231661 2563 server.go:1268] "Started kubelet" Jul 21 08:18:48.231764 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.231741 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 08:18:48.232225 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.232199 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 08:18:48.232304 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.232241 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 08:18:48.232577 ip-10-0-137-36 systemd[1]: Started Kubernetes Kubelet. Jul 21 08:18:48.233628 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.233599 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 08:18:48.233628 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.233607 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 08:18:48.234599 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.234579 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 08:18:48.234768 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.234585 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-36.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 08:18:48.235792 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.235771 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 21 08:18:48.237481 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.237462 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 08:18:48.237815 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.236986 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-36.ec2.internal.18c440303a9e834b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-36.ec2.internal,UID:ip-10-0-137-36.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-36.ec2.internal,},FirstTimestamp:2026-07-21 08:18:48.231633739 +0000 UTC m=+0.310672062,LastTimestamp:2026-07-21 08:18:48.231633739 +0000 UTC m=+0.310672062,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-36.ec2.internal,}" Jul 21 08:18:48.241453 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.241440 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 08:18:48.241503 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.241477 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 08:18:48.241564 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.241515 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:48.242003 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.241981 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 08:18:48.242003 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242002 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 08:18:48.242115 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242014 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 08:18:48.242171 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242126 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 21 08:18:48.242171 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242134 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 21 08:18:48.242577 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242559 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 08:18:48.242577 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242577 2563 factory.go:55] Registering systemd factory Jul 21 08:18:48.242698 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242586 2563 factory.go:223] Registration of the systemd container factory successfully Jul 21 08:18:48.242827 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242811 2563 factory.go:153] Registering CRI-O factory Jul 21 08:18:48.242827 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242827 2563 factory.go:223] Registration of the crio container factory successfully Jul 21 08:18:48.242946 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242847 2563 factory.go:103] Registering Raw factory Jul 21 08:18:48.242946 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.242856 2563 manager.go:1196] Started watching for new ooms in manager Jul 21 08:18:48.243162 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.243152 2563 manager.go:349] Starting recovery of all containers Jul 21 08:18:48.243556 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.243533 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 08:18:48.243659 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.243642 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 08:18:48.252704 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.252687 2563 manager.go:354] Recovery completed Jul 21 08:18:48.255216 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.255197 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mm8cp" Jul 21 08:18:48.256801 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.256785 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:18:48.258501 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.258485 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:18:48.258581 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.258516 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:18:48.258581 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.258527 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:18:48.259056 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.259039 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 08:18:48.259056 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.259053 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 08:18:48.259167 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.259070 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 21 08:18:48.259167 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.259055 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:18:48.259782 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.259765 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:18:48.259848 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.259794 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:18:48.259848 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.259805 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:18:48.260276 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.260261 2563 policy_none.go:49] "None policy: Start" Jul 21 08:18:48.260351 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.260279 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 08:18:48.260351 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.260289 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 08:18:48.261354 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.261344 2563 policy_none.go:47] "Start" Jul 21 08:18:48.262574 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.262551 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mm8cp" Jul 21 08:18:48.267019 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.266954 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-36.ec2.internal.18c440303c3877ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-36.ec2.internal,UID:ip-10-0-137-36.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-36.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-36.ec2.internal,},FirstTimestamp:2026-07-21 08:18:48.258500523 +0000 UTC m=+0.337538853,LastTimestamp:2026-07-21 08:18:48.258500523 +0000 UTC m=+0.337538853,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-36.ec2.internal,}" Jul 21 08:18:48.302939 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.302923 2563 manager.go:337] "Starting Device Plugin manager" Jul 21 08:18:48.303006 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.302955 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 08:18:48.303006 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.302969 2563 server.go:87] "Starting device plugin registration server" Jul 21 08:18:48.303231 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.303220 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 08:18:48.303313 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.303232 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 08:18:48.303527 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.303332 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 08:18:48.303617 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.303603 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 08:18:48.303617 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.303612 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 08:18:48.304016 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.303999 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 08:18:48.304080 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.304049 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:48.350116 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.350083 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 08:18:48.351378 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.351361 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 08:18:48.351449 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.351430 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 08:18:48.351483 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.351458 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 08:18:48.351513 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.351499 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 08:18:48.353830 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.353812 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:18:48.403766 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.403739 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:18:48.405527 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.405508 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:18:48.405618 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.405541 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:18:48.405618 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.405553 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:18:48.405618 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.405575 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.414503 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.414438 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.414503 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.414461 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-36.ec2.internal\": node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:48.441895 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.441872 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:48.451580 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.451559 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal"] Jul 21 08:18:48.451666 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.451614 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:18:48.452505 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.452490 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:18:48.452579 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.452517 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:18:48.452579 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.452526 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:18:48.453749 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.453735 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:18:48.453871 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.453857 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.453934 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.453907 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:18:48.454445 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.454431 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:18:48.454531 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.454464 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:18:48.454531 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.454478 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:18:48.454531 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.454434 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:18:48.454673 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.454537 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:18:48.454673 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.454549 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:18:48.455720 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.455703 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.455805 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.455732 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:18:48.456401 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.456372 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:18:48.456488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.456408 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:18:48.456488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.456424 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:18:48.469914 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.469895 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-36.ec2.internal\" not found" node="ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.473668 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.473653 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-36.ec2.internal\" not found" node="ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.542911 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.542887 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:48.643254 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.643223 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:48.643418 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.643285 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7bcbd72e8164ffd2b41df6d71e8e46c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal\" (UID: \"e7bcbd72e8164ffd2b41df6d71e8e46c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.643418 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.643308 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bcbd72e8164ffd2b41df6d71e8e46c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal\" (UID: \"e7bcbd72e8164ffd2b41df6d71e8e46c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.643418 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.643328 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562d6f5337e483c37d48c2d1b5b30966-config\") pod \"kube-apiserver-proxy-ip-10-0-137-36.ec2.internal\" (UID: \"562d6f5337e483c37d48c2d1b5b30966\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.743907 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.743834 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:48.743907 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.743869 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562d6f5337e483c37d48c2d1b5b30966-config\") pod \"kube-apiserver-proxy-ip-10-0-137-36.ec2.internal\" (UID: \"562d6f5337e483c37d48c2d1b5b30966\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.743907 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.743894 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7bcbd72e8164ffd2b41df6d71e8e46c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal\" (UID: \"e7bcbd72e8164ffd2b41df6d71e8e46c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.744084 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.743911 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bcbd72e8164ffd2b41df6d71e8e46c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal\" (UID: \"e7bcbd72e8164ffd2b41df6d71e8e46c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.744084 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.743951 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7bcbd72e8164ffd2b41df6d71e8e46c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal\" (UID: \"e7bcbd72e8164ffd2b41df6d71e8e46c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.744084 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.743957 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562d6f5337e483c37d48c2d1b5b30966-config\") pod \"kube-apiserver-proxy-ip-10-0-137-36.ec2.internal\" (UID: \"562d6f5337e483c37d48c2d1b5b30966\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.744084 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.743992 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bcbd72e8164ffd2b41df6d71e8e46c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal\" (UID: \"e7bcbd72e8164ffd2b41df6d71e8e46c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.774056 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.774022 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.776374 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:48.776357 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal" Jul 21 08:18:48.844362 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.844332 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:48.944696 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:48.944670 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:49.045195 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:49.045123 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:49.145526 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:49.145479 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:49.181683 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.181653 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 08:18:49.182110 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.181780 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 08:18:49.245618 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:49.245587 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:49.267086 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.267019 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 08:13:48 +0000 UTC" deadline="2027-12-23 15:49:28.106332816 +0000 UTC" Jul 21 08:18:49.267086 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.267058 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12487h30m38.83927973s" Jul 21 08:18:49.270412 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.270387 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:18:49.286783 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.286747 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:18:49.288128 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.288103 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:18:49.288229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.288147 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:18:49.288229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.288160 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:18:49.345949 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:49.345930 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:49.355430 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:49.355383 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562d6f5337e483c37d48c2d1b5b30966.slice/crio-8dd0f3d1d6da9a3475dd66815e4dfee92055f78a67e3d3f1c1f95275d6899a9e WatchSource:0}: Error finding container 8dd0f3d1d6da9a3475dd66815e4dfee92055f78a67e3d3f1c1f95275d6899a9e: Status 404 returned error can't find the container with id 8dd0f3d1d6da9a3475dd66815e4dfee92055f78a67e3d3f1c1f95275d6899a9e Jul 21 08:18:49.356108 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:49.356073 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bcbd72e8164ffd2b41df6d71e8e46c.slice/crio-b264905b1e83dc1a2b0792f0bc45b00cea18c8eef1526a1d458b555959dd954a WatchSource:0}: Error finding container b264905b1e83dc1a2b0792f0bc45b00cea18c8eef1526a1d458b555959dd954a: Status 404 returned error can't find the container with id b264905b1e83dc1a2b0792f0bc45b00cea18c8eef1526a1d458b555959dd954a Jul 21 08:18:49.358169 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.358149 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:18:49.361432 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.361419 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:18:49.446574 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:49.446518 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:49.547263 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:49.547224 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-36.ec2.internal\" not found" Jul 21 08:18:49.587346 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.587320 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:18:49.645721 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.645633 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" Jul 21 08:18:49.660457 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.660433 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 08:18:49.661387 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.661373 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal" Jul 21 08:18:49.670220 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:49.670203 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 08:18:50.223826 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.223542 2563 apiserver.go:52] "Watching apiserver" Jul 21 08:18:50.231655 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.231626 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 08:18:50.232270 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.232244 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-7dgc9","openshift-network-diagnostics/network-check-target-88msb","kube-system/global-pull-secret-syncer-fkhp9","kube-system/konnectivity-agent-m5l9v","kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms","openshift-cluster-node-tuning-operator/tuned-bwlz9","openshift-image-registry/node-ca-h5m66","openshift-network-operator/iptables-alerter-9n8qb","openshift-ovn-kubernetes/ovnkube-node-q6tbn","openshift-dns/node-resolver-llppt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal","openshift-multus/multus-additional-cni-plugins-rvbx6","openshift-multus/multus-ptcvr"] Jul 21 08:18:50.234066 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.234043 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.235911 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.235884 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.236569 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.236545 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 08:18:50.236675 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.236622 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pncr8\"" Jul 21 08:18:50.236732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.236545 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 08:18:50.240367 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.240345 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 08:18:50.241553 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.240548 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.241553 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.240599 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x5ftn\"" Jul 21 08:18:50.241553 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.240619 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:18:50.241553 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.240829 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 08:18:50.241553 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.240999 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:18:50.241553 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.241091 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.241900 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.241627 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 08:18:50.242824 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.242756 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.243424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.243397 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 08:18:50.243842 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.243824 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 08:18:50.243953 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.243935 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 08:18:50.244130 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.244113 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 08:18:50.244195 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.244179 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 08:18:50.244258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.244113 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 08:18:50.244727 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.244710 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-52k75\"" Jul 21 08:18:50.244826 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.244710 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b6gpp\"" Jul 21 08:18:50.245283 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.245250 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 08:18:50.245643 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.245622 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jb2fk\"" Jul 21 08:18:50.245854 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.245836 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 08:18:50.246869 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.246130 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.246967 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.246956 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 08:18:50.247916 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.247570 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.248594 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.248570 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 08:18:50.248710 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.248676 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 08:18:50.248855 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.248837 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 08:18:50.248991 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.248973 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vztbx\"" Jul 21 08:18:50.249156 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.249139 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:50.249230 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.249203 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:18:50.250495 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.249986 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h9khx\"" Jul 21 08:18:50.250495 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.250244 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.254768 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255083 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59jh\" (UniqueName: \"kubernetes.io/projected/a59731f0-2456-4447-a672-69a180edc896-kube-api-access-n59jh\") pod \"node-resolver-llppt\" (UID: \"a59731f0-2456-4447-a672-69a180edc896\") " pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255122 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-run-k8s-cni-cncf-io\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-daemon-config\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255199 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-sys-fs\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255232 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-sys\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-lib-modules\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255319 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-host\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255360 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-conf-dir\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255416 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-var-lib-kubelet\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.255452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255475 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-etc-selinux\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255500 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-tuned\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255593 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59731f0-2456-4447-a672-69a180edc896-tmp-dir\") pod \"node-resolver-llppt\" (UID: \"a59731f0-2456-4447-a672-69a180edc896\") " pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255648 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-kubelet-config\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255690 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-dbus\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255716 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-cnibin\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255741 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255771 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-socket-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255806 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-sysctl-conf\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255829 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a59731f0-2456-4447-a672-69a180edc896-hosts-file\") pod \"node-resolver-llppt\" (UID: \"a59731f0-2456-4447-a672-69a180edc896\") " pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255917 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39eb937b-c6b2-47bf-97ab-7ccd168023ec-konnectivity-ca\") pod \"konnectivity-agent-m5l9v\" (UID: \"39eb937b-c6b2-47bf-97ab-7ccd168023ec\") " pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255940 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrsxs\" (UniqueName: \"kubernetes.io/projected/e55e17f4-78a0-4e21-ab12-0ce0471ec960-kube-api-access-rrsxs\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-systemd\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.255995 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-cni-dir\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256028 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-socket-dir-parent\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256054 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-kubernetes\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.256098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256078 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-run\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256158 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-run-multus-certs\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256184 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-etc-kubernetes\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256219 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39eb937b-c6b2-47bf-97ab-7ccd168023ec-agent-certs\") pod \"konnectivity-agent-m5l9v\" (UID: \"39eb937b-c6b2-47bf-97ab-7ccd168023ec\") " pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256253 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-system-cni-dir\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256303 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-run-netns\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256340 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-hostroot\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256504 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtgx\" (UniqueName: \"kubernetes.io/projected/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-kube-api-access-4gtgx\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256528 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-cni-binary-copy\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256566 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-var-lib-cni-bin\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256601 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-var-lib-cni-multus\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256672 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-registration-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256713 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-device-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256743 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-modprobe-d\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256837 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-sysconfig\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.256962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.256918 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-var-lib-kubelet\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.257733 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.257098 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-os-release\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.257733 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.257142 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-sysctl-d\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.257733 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.257182 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/988708bc-ebff-4e44-9fd5-fde5f19594ed-tmp\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.257733 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.257256 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.257733 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.257289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbzmr\" (UniqueName: \"kubernetes.io/projected/988708bc-ebff-4e44-9fd5-fde5f19594ed-kube-api-access-lbzmr\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.257733 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.257359 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:50.257733 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.257453 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:18:50.259092 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.259071 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.261276 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.261255 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 08:18:50.261563 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.261545 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f4fp4\"" Jul 21 08:18:50.261787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.261772 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 08:18:50.261937 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.261553 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 08:18:50.262149 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.262126 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 08:18:50.262226 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.262150 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 08:18:50.262648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.262336 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 08:18:50.262648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.262391 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 08:18:50.264088 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.263710 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 08:18:50.264088 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.263780 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 08:18:50.264088 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.263937 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhbf2\"" Jul 21 08:18:50.283558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.283535 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vh7fw" Jul 21 08:18:50.284921 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.284895 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:18:50.291167 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.291146 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vh7fw" Jul 21 08:18:50.343322 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.343297 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 08:18:50.357489 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357399 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal" event={"ID":"562d6f5337e483c37d48c2d1b5b30966","Type":"ContainerStarted","Data":"8dd0f3d1d6da9a3475dd66815e4dfee92055f78a67e3d3f1c1f95275d6899a9e"} Jul 21 08:18:50.357655 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357586 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-run-systemd\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.357655 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357621 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-env-overrides\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.357839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59731f0-2456-4447-a672-69a180edc896-tmp-dir\") pod \"node-resolver-llppt\" (UID: \"a59731f0-2456-4447-a672-69a180edc896\") " pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.357839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357682 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-cnibin\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.357839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357708 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.357839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357747 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-log-socket\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.357839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-ovnkube-config\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.357839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357805 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-systemd\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357838 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdw47\" (UniqueName: \"kubernetes.io/projected/3c68db5d-94fa-4001-ad5a-985f8f568fce-kube-api-access-vdw47\") pod \"iptables-alerter-9n8qb\" (UID: \"3c68db5d-94fa-4001-ad5a-985f8f568fce\") " pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357868 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-cni-dir\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357893 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-socket-dir-parent\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357918 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-kubernetes\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357924 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357940 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-run\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.357983 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c044a2bf-898d-4e0c-b624-01d347f5d5f9-serviceca\") pod \"node-ca-h5m66\" (UID: \"c044a2bf-898d-4e0c-b624-01d347f5d5f9\") " pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358009 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-var-lib-openvswitch\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358087 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-cnibin\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-etc-kubernetes\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358150 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-system-cni-dir\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358177 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-run-netns\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtgx\" (UniqueName: \"kubernetes.io/projected/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-kube-api-access-4gtgx\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358244 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-device-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358260 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59731f0-2456-4447-a672-69a180edc896-tmp-dir\") pod \"node-resolver-llppt\" (UID: \"a59731f0-2456-4447-a672-69a180edc896\") " pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358282 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-etc-openvswitch\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358311 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs2md\" (UniqueName: \"kubernetes.io/projected/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-kube-api-access-cs2md\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358337 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-cni-binary-copy\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358351 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-cnibin\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358370 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-var-lib-cni-bin\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-registration-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358425 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/988708bc-ebff-4e44-9fd5-fde5f19594ed-tmp\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358479 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbzmr\" (UniqueName: \"kubernetes.io/projected/988708bc-ebff-4e44-9fd5-fde5f19594ed-kube-api-access-lbzmr\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358521 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58g2\" (UniqueName: \"kubernetes.io/projected/c044a2bf-898d-4e0c-b624-01d347f5d5f9-kube-api-access-x58g2\") pod \"node-ca-h5m66\" (UID: \"c044a2bf-898d-4e0c-b624-01d347f5d5f9\") " pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358547 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2d8n\" (UniqueName: \"kubernetes.io/projected/25959877-ace6-4180-9de6-f2bd857a8ff7-kube-api-access-b2d8n\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358576 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-os-release\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.358610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358595 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-etc-kubernetes\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358658 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-sysctl-d\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358692 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-slash\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358703 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-systemd\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358730 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-run-netns\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358761 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-ovnkube-script-lib\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358787 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-daemon-config\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358821 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-sys-fs\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358847 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-lib-modules\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-kubelet\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358895 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-systemd-units\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-run-ovn\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358931 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-cni-dir\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358961 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-conf-dir\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-socket-dir-parent\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.358991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-etc-selinux\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-tuned\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.359385 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359057 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-kubernetes\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359071 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-cni-bin\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359107 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-cni-netd\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359112 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-run\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359151 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-kubelet-config\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-dbus\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359280 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-dbus\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359323 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-system-cni-dir\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359397 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-run-netns\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359661 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-var-lib-cni-bin\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359730 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-registration-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359791 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-device-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359862 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-socket-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-sysctl-conf\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-node-log\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359970 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360027 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a59731f0-2456-4447-a672-69a180edc896-hosts-file\") pod \"node-resolver-llppt\" (UID: \"a59731f0-2456-4447-a672-69a180edc896\") " pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.360252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360056 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39eb937b-c6b2-47bf-97ab-7ccd168023ec-konnectivity-ca\") pod \"konnectivity-agent-m5l9v\" (UID: \"39eb937b-c6b2-47bf-97ab-7ccd168023ec\") " pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360081 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-cni-binary-copy\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrsxs\" (UniqueName: \"kubernetes.io/projected/e55e17f4-78a0-4e21-ab12-0ce0471ec960-kube-api-access-rrsxs\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.359921 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-conf-dir\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360375 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a59731f0-2456-4447-a672-69a180edc896-hosts-file\") pod \"node-resolver-llppt\" (UID: \"a59731f0-2456-4447-a672-69a180edc896\") " pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360391 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-etc-selinux\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360478 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-os-release\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360590 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-sysctl-d\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360583 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-multus-daemon-config\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360603 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-sys-fs\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360648 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-ovn-node-metrics-cert\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360714 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c044a2bf-898d-4e0c-b624-01d347f5d5f9-host\") pod \"node-ca-h5m66\" (UID: \"c044a2bf-898d-4e0c-b624-01d347f5d5f9\") " pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360691 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-sysctl-conf\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360849 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360875 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-lib-modules\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.360948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360887 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e55e17f4-78a0-4e21-ab12-0ce0471ec960-socket-dir\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.360966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c68db5d-94fa-4001-ad5a-985f8f568fce-iptables-alerter-script\") pod \"iptables-alerter-9n8qb\" (UID: \"3c68db5d-94fa-4001-ad5a-985f8f568fce\") " pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c68db5d-94fa-4001-ad5a-985f8f568fce-host-slash\") pod \"iptables-alerter-9n8qb\" (UID: \"3c68db5d-94fa-4001-ad5a-985f8f568fce\") " pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361070 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25959877-ace6-4180-9de6-f2bd857a8ff7-cni-binary-copy\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361107 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361133 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361202 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-run-multus-certs\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39eb937b-c6b2-47bf-97ab-7ccd168023ec-agent-certs\") pod \"konnectivity-agent-m5l9v\" (UID: \"39eb937b-c6b2-47bf-97ab-7ccd168023ec\") " pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361235 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39eb937b-c6b2-47bf-97ab-7ccd168023ec-konnectivity-ca\") pod \"konnectivity-agent-m5l9v\" (UID: \"39eb937b-c6b2-47bf-97ab-7ccd168023ec\") " pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361251 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-hostroot\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361277 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-sysconfig\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361283 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-run-multus-certs\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361340 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-run-openvswitch\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361372 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361368 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-hostroot\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361430 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhhk\" (UniqueName: \"kubernetes.io/projected/19a14646-a2c6-4a63-8716-6fb7127293cf-kube-api-access-6vhhk\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:50.361747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361447 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-sysconfig\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.361586 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-var-lib-cni-multus\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363577 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-var-lib-cni-multus\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363574 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-kubelet-config\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363640 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-modprobe-d\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363703 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-var-lib-kubelet\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363746 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-modprobe-d\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363752 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25959877-ace6-4180-9de6-f2bd857a8ff7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-os-release\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363822 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-var-lib-kubelet\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363858 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25959877-ace6-4180-9de6-f2bd857a8ff7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363875 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" event={"ID":"e7bcbd72e8164ffd2b41df6d71e8e46c","Type":"ContainerStarted","Data":"b264905b1e83dc1a2b0792f0bc45b00cea18c8eef1526a1d458b555959dd954a"} Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363899 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59jh\" (UniqueName: \"kubernetes.io/projected/a59731f0-2456-4447-a672-69a180edc896-kube-api-access-n59jh\") pod \"node-resolver-llppt\" (UID: \"a59731f0-2456-4447-a672-69a180edc896\") " pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-run-k8s-cni-cncf-io\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.363982 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-sys\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.364009 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-host\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.364043 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-system-cni-dir\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.365744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.364081 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.366607 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.364064 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-run-k8s-cni-cncf-io\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.366607 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.364166 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-var-lib-kubelet\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.366607 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.364495 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-sys\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.366607 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.364115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-host-var-lib-kubelet\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.366607 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.364563 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/988708bc-ebff-4e44-9fd5-fde5f19594ed-host\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.366607 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.364667 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:50.366607 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.366011 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/988708bc-ebff-4e44-9fd5-fde5f19594ed-tmp\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.366607 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.366424 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39eb937b-c6b2-47bf-97ab-7ccd168023ec-agent-certs\") pod \"konnectivity-agent-m5l9v\" (UID: \"39eb937b-c6b2-47bf-97ab-7ccd168023ec\") " pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:18:50.368146 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.367959 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret podName:ae3fb7e3-9363-4e30-b5c4-362303bd14bb nodeName:}" failed. No retries permitted until 2026-07-21 08:18:50.867910665 +0000 UTC m=+2.946948990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret") pod "global-pull-secret-syncer-fkhp9" (UID: "ae3fb7e3-9363-4e30-b5c4-362303bd14bb") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:50.368283 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.368152 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/988708bc-ebff-4e44-9fd5-fde5f19594ed-etc-tuned\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.373964 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.373940 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrsxs\" (UniqueName: \"kubernetes.io/projected/e55e17f4-78a0-4e21-ab12-0ce0471ec960-kube-api-access-rrsxs\") pod \"aws-ebs-csi-driver-node-gl8ms\" (UID: \"e55e17f4-78a0-4e21-ab12-0ce0471ec960\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.384918 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.384889 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59jh\" (UniqueName: \"kubernetes.io/projected/a59731f0-2456-4447-a672-69a180edc896-kube-api-access-n59jh\") pod \"node-resolver-llppt\" (UID: \"a59731f0-2456-4447-a672-69a180edc896\") " pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.385448 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.385425 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtgx\" (UniqueName: \"kubernetes.io/projected/d77c79ae-c248-4c1f-9a2b-b4b63460b83f-kube-api-access-4gtgx\") pod \"multus-ptcvr\" (UID: \"d77c79ae-c248-4c1f-9a2b-b4b63460b83f\") " pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.386480 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.386455 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbzmr\" (UniqueName: \"kubernetes.io/projected/988708bc-ebff-4e44-9fd5-fde5f19594ed-kube-api-access-lbzmr\") pod \"tuned-bwlz9\" (UID: \"988708bc-ebff-4e44-9fd5-fde5f19594ed\") " pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.468833 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.468794 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-slash\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469018 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.468845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-run-netns\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469018 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.468904 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-run-netns\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469018 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.468919 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-slash\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469197 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-ovnkube-script-lib\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469197 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-kubelet\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469197 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-systemd-units\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469197 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-run-ovn\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469197 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469127 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-cni-bin\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469197 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469143 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-kubelet\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469202 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-cni-netd\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-cni-netd\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469243 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-systemd-units\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469252 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-node-log\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469284 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469311 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-run-ovn\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469345 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-ovn-node-metrics-cert\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469361 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-node-log\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469370 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c044a2bf-898d-4e0c-b624-01d347f5d5f9-host\") pod \"node-ca-h5m66\" (UID: \"c044a2bf-898d-4e0c-b624-01d347f5d5f9\") " pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469400 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c68db5d-94fa-4001-ad5a-985f8f568fce-iptables-alerter-script\") pod \"iptables-alerter-9n8qb\" (UID: \"3c68db5d-94fa-4001-ad5a-985f8f568fce\") " pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469447 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c68db5d-94fa-4001-ad5a-985f8f568fce-host-slash\") pod \"iptables-alerter-9n8qb\" (UID: \"3c68db5d-94fa-4001-ad5a-985f8f568fce\") " pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469473 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c044a2bf-898d-4e0c-b624-01d347f5d5f9-host\") pod \"node-ca-h5m66\" (UID: \"c044a2bf-898d-4e0c-b624-01d347f5d5f9\") " pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25959877-ace6-4180-9de6-f2bd857a8ff7-cni-binary-copy\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.469504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469528 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469549 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-run-openvswitch\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469566 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469564 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469584 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhhk\" (UniqueName: \"kubernetes.io/projected/19a14646-a2c6-4a63-8716-6fb7127293cf-kube-api-access-6vhhk\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469632 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25959877-ace6-4180-9de6-f2bd857a8ff7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469677 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-os-release\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469704 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25959877-ace6-4180-9de6-f2bd857a8ff7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469709 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-ovnkube-script-lib\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469737 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-system-cni-dir\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469779 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-run-systemd\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469804 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-env-overrides\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469834 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-log-socket\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-ovnkube-config\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469865 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c68db5d-94fa-4001-ad5a-985f8f568fce-host-slash\") pod \"iptables-alerter-9n8qb\" (UID: \"3c68db5d-94fa-4001-ad5a-985f8f568fce\") " pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469890 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdw47\" (UniqueName: \"kubernetes.io/projected/3c68db5d-94fa-4001-ad5a-985f8f568fce-kube-api-access-vdw47\") pod \"iptables-alerter-9n8qb\" (UID: \"3c68db5d-94fa-4001-ad5a-985f8f568fce\") " pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.470258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c044a2bf-898d-4e0c-b624-01d347f5d5f9-serviceca\") pod \"node-ca-h5m66\" (UID: \"c044a2bf-898d-4e0c-b624-01d347f5d5f9\") " pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469951 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-var-lib-openvswitch\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469977 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-cnibin\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470009 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-etc-openvswitch\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs2md\" (UniqueName: \"kubernetes.io/projected/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-kube-api-access-cs2md\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470054 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c68db5d-94fa-4001-ad5a-985f8f568fce-iptables-alerter-script\") pod \"iptables-alerter-9n8qb\" (UID: \"3c68db5d-94fa-4001-ad5a-985f8f568fce\") " pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469361 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-cni-bin\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470085 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58g2\" (UniqueName: \"kubernetes.io/projected/c044a2bf-898d-4e0c-b624-01d347f5d5f9-kube-api-access-x58g2\") pod \"node-ca-h5m66\" (UID: \"c044a2bf-898d-4e0c-b624-01d347f5d5f9\") " pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470131 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-log-socket\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2d8n\" (UniqueName: \"kubernetes.io/projected/25959877-ace6-4180-9de6-f2bd857a8ff7-kube-api-access-b2d8n\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470320 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-var-lib-openvswitch\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470374 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-cnibin\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470487 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-etc-openvswitch\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25959877-ace6-4180-9de6-f2bd857a8ff7-cni-binary-copy\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470789 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-ovnkube-config\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.470938 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-system-cni-dir\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.471014 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25959877-ace6-4180-9de6-f2bd857a8ff7-os-release\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.471152 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.471053 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:50.471968 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.471128 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:18:50.971110128 +0000 UTC m=+3.050148457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:50.471968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.471136 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c044a2bf-898d-4e0c-b624-01d347f5d5f9-serviceca\") pod \"node-ca-h5m66\" (UID: \"c044a2bf-898d-4e0c-b624-01d347f5d5f9\") " pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.471968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.471187 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.471222 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-run-openvswitch\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.471049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-run-systemd\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.471429 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-env-overrides\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.469398 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.471968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.471575 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25959877-ace6-4180-9de6-f2bd857a8ff7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.472370 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.472037 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25959877-ace6-4180-9de6-f2bd857a8ff7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.472425 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.472406 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-ovn-node-metrics-cert\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.481552 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.481483 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:18:50.481552 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.481511 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:18:50.481552 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.481524 2563 projected.go:196] Error preparing data for projected volume kube-api-access-x42cd for pod openshift-network-diagnostics/network-check-target-88msb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:50.481781 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.481600 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd podName:131059bc-6588-40c7-a8c5-73664c7e59e4 nodeName:}" failed. No retries permitted until 2026-07-21 08:18:50.981581458 +0000 UTC m=+3.060619786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x42cd" (UniqueName: "kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd") pod "network-check-target-88msb" (UID: "131059bc-6588-40c7-a8c5-73664c7e59e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:50.483852 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.483826 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs2md\" (UniqueName: \"kubernetes.io/projected/9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d-kube-api-access-cs2md\") pod \"ovnkube-node-q6tbn\" (UID: \"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.484189 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.484170 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58g2\" (UniqueName: \"kubernetes.io/projected/c044a2bf-898d-4e0c-b624-01d347f5d5f9-kube-api-access-x58g2\") pod \"node-ca-h5m66\" (UID: \"c044a2bf-898d-4e0c-b624-01d347f5d5f9\") " pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.484344 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.484322 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2d8n\" (UniqueName: \"kubernetes.io/projected/25959877-ace6-4180-9de6-f2bd857a8ff7-kube-api-access-b2d8n\") pod \"multus-additional-cni-plugins-rvbx6\" (UID: \"25959877-ace6-4180-9de6-f2bd857a8ff7\") " pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.484826 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.484806 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhhk\" (UniqueName: \"kubernetes.io/projected/19a14646-a2c6-4a63-8716-6fb7127293cf-kube-api-access-6vhhk\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:50.485186 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.485169 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdw47\" (UniqueName: \"kubernetes.io/projected/3c68db5d-94fa-4001-ad5a-985f8f568fce-kube-api-access-vdw47\") pod \"iptables-alerter-9n8qb\" (UID: \"3c68db5d-94fa-4001-ad5a-985f8f568fce\") " pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.555200 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.555163 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" Jul 21 08:18:50.565368 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.565341 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-llppt" Jul 21 08:18:50.576121 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.576090 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:18:50.582081 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.582055 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ptcvr" Jul 21 08:18:50.590872 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.590853 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" Jul 21 08:18:50.600616 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.600595 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9n8qb" Jul 21 08:18:50.609522 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.609503 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" Jul 21 08:18:50.617300 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.617277 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:18:50.627104 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.627084 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5m66" Jul 21 08:18:50.873227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.873142 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:50.873384 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.873327 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:50.873463 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.873430 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret podName:ae3fb7e3-9363-4e30-b5c4-362303bd14bb nodeName:}" failed. No retries permitted until 2026-07-21 08:18:51.873390612 +0000 UTC m=+3.952428923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret") pod "global-pull-secret-syncer-fkhp9" (UID: "ae3fb7e3-9363-4e30-b5c4-362303bd14bb") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:50.974119 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:50.974076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:50.974286 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.974212 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:50.974286 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:50.974272 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:18:51.974256376 +0000 UTC m=+4.053294686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:51.074671 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:51.074630 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:51.074859 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:51.074828 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:18:51.074859 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:51.074854 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:18:51.074974 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:51.074864 2563 projected.go:196] Error preparing data for projected volume kube-api-access-x42cd for pod openshift-network-diagnostics/network-check-target-88msb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:51.074974 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:51.074931 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd podName:131059bc-6588-40c7-a8c5-73664c7e59e4 nodeName:}" failed. No retries permitted until 2026-07-21 08:18:52.074915429 +0000 UTC m=+4.153953764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42cd" (UniqueName: "kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd") pod "network-check-target-88msb" (UID: "131059bc-6588-40c7-a8c5-73664c7e59e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:51.293663 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:51.293624 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 08:13:50 +0000 UTC" deadline="2028-03-15 11:57:45.583324013 +0000 UTC" Jul 21 08:18:51.293663 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:51.293655 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14475h38m54.289672574s" Jul 21 08:18:51.548017 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:51.547987 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59731f0_2456_4447_a672_69a180edc896.slice/crio-eaf3d2285c8e40d2a087215ad636e7111f575210cfa6bf462a69e67a1795f110 WatchSource:0}: Error finding container eaf3d2285c8e40d2a087215ad636e7111f575210cfa6bf462a69e67a1795f110: Status 404 returned error can't find the container with id eaf3d2285c8e40d2a087215ad636e7111f575210cfa6bf462a69e67a1795f110 Jul 21 08:18:51.549625 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:51.549587 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39eb937b_c6b2_47bf_97ab_7ccd168023ec.slice/crio-b35165bc859c0ccd0ba30686063cd9d67c6884727105e5fbd264496b73475fbb WatchSource:0}: Error finding container b35165bc859c0ccd0ba30686063cd9d67c6884727105e5fbd264496b73475fbb: Status 404 returned error can't find the container with id b35165bc859c0ccd0ba30686063cd9d67c6884727105e5fbd264496b73475fbb Jul 21 08:18:51.550378 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:51.550352 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988708bc_ebff_4e44_9fd5_fde5f19594ed.slice/crio-92a02ff071cc292d3e17f6fc9a0c8aa2a6a8b2870bf260e5bebe3b7d0622c9a9 WatchSource:0}: Error finding container 92a02ff071cc292d3e17f6fc9a0c8aa2a6a8b2870bf260e5bebe3b7d0622c9a9: Status 404 returned error can't find the container with id 92a02ff071cc292d3e17f6fc9a0c8aa2a6a8b2870bf260e5bebe3b7d0622c9a9 Jul 21 08:18:51.551469 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:51.551371 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9bec6d_5a3a_45a7_be1a_e5aa116ce18d.slice/crio-5093ef49515eb6aaf77a3be841a2a9ebd425db4d6c32d192e146213c7628ab6a WatchSource:0}: Error finding container 5093ef49515eb6aaf77a3be841a2a9ebd425db4d6c32d192e146213c7628ab6a: Status 404 returned error can't find the container with id 5093ef49515eb6aaf77a3be841a2a9ebd425db4d6c32d192e146213c7628ab6a Jul 21 08:18:51.552633 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:51.552514 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc044a2bf_898d_4e0c_b624_01d347f5d5f9.slice/crio-e1d9cc4c2a71b0a5eada9fff4e75e0a1ede8f815caaf4b30870c446db0a4e3fd WatchSource:0}: Error finding container e1d9cc4c2a71b0a5eada9fff4e75e0a1ede8f815caaf4b30870c446db0a4e3fd: Status 404 returned error can't find the container with id e1d9cc4c2a71b0a5eada9fff4e75e0a1ede8f815caaf4b30870c446db0a4e3fd Jul 21 08:18:51.553865 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:51.553841 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25959877_ace6_4180_9de6_f2bd857a8ff7.slice/crio-74b2ed22ea5b482d3049f5238afcabecd16ab8d5073a1f11fe65e19903349ecb WatchSource:0}: Error finding container 74b2ed22ea5b482d3049f5238afcabecd16ab8d5073a1f11fe65e19903349ecb: Status 404 returned error can't find the container with id 74b2ed22ea5b482d3049f5238afcabecd16ab8d5073a1f11fe65e19903349ecb Jul 21 08:18:51.555015 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:51.554946 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c68db5d_94fa_4001_ad5a_985f8f568fce.slice/crio-2f50c827ddbd80e6318b8eeaf098eced6d84fa402a6abf0070cd9425e9e123bd WatchSource:0}: Error finding container 2f50c827ddbd80e6318b8eeaf098eced6d84fa402a6abf0070cd9425e9e123bd: Status 404 returned error can't find the container with id 2f50c827ddbd80e6318b8eeaf098eced6d84fa402a6abf0070cd9425e9e123bd Jul 21 08:18:51.557317 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:18:51.557290 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77c79ae_c248_4c1f_9a2b_b4b63460b83f.slice/crio-c3b62d157734bb38bc55d58970bda6437e4d28bb7332bf7f8b97f8ad509ca2b5 WatchSource:0}: Error finding container c3b62d157734bb38bc55d58970bda6437e4d28bb7332bf7f8b97f8ad509ca2b5: Status 404 returned error can't find the container with id c3b62d157734bb38bc55d58970bda6437e4d28bb7332bf7f8b97f8ad509ca2b5 Jul 21 08:18:51.882160 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:51.881969 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:51.882316 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:51.882130 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:51.882316 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:51.882234 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret podName:ae3fb7e3-9363-4e30-b5c4-362303bd14bb nodeName:}" failed. No retries permitted until 2026-07-21 08:18:53.882213323 +0000 UTC m=+5.961251634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret") pod "global-pull-secret-syncer-fkhp9" (UID: "ae3fb7e3-9363-4e30-b5c4-362303bd14bb") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:51.982860 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:51.982818 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:51.983030 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:51.982933 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:51.983030 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:51.982989 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:18:53.98297412 +0000 UTC m=+6.062012431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:52.084527 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.083989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:52.084527 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:52.084283 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:18:52.084527 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:52.084306 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:18:52.084527 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:52.084321 2563 projected.go:196] Error preparing data for projected volume kube-api-access-x42cd for pod openshift-network-diagnostics/network-check-target-88msb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:52.084527 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:52.084383 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd podName:131059bc-6588-40c7-a8c5-73664c7e59e4 nodeName:}" failed. No retries permitted until 2026-07-21 08:18:54.084364985 +0000 UTC m=+6.163403309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42cd" (UniqueName: "kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd") pod "network-check-target-88msb" (UID: "131059bc-6588-40c7-a8c5-73664c7e59e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:52.296690 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.296618 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 08:13:50 +0000 UTC" deadline="2027-12-17 11:29:12.13515575 +0000 UTC" Jul 21 08:18:52.296690 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.296688 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12339h10m19.838471968s" Jul 21 08:18:52.352629 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.352552 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:52.352796 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:52.352688 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:18:52.353124 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.353101 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:52.353317 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:52.353229 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:18:52.353317 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.353298 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:52.353442 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:52.353369 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:18:52.372942 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.372870 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" event={"ID":"e55e17f4-78a0-4e21-ab12-0ce0471ec960","Type":"ContainerStarted","Data":"fba2832bdf2b9a923dbfe4104d15697ccfc828720031cdf35ef69c5037ec5b7d"} Jul 21 08:18:52.377373 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.377340 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9n8qb" event={"ID":"3c68db5d-94fa-4001-ad5a-985f8f568fce","Type":"ContainerStarted","Data":"2f50c827ddbd80e6318b8eeaf098eced6d84fa402a6abf0070cd9425e9e123bd"} Jul 21 08:18:52.389169 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.389131 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" event={"ID":"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d","Type":"ContainerStarted","Data":"5093ef49515eb6aaf77a3be841a2a9ebd425db4d6c32d192e146213c7628ab6a"} Jul 21 08:18:52.400449 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.400417 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5l9v" event={"ID":"39eb937b-c6b2-47bf-97ab-7ccd168023ec","Type":"ContainerStarted","Data":"b35165bc859c0ccd0ba30686063cd9d67c6884727105e5fbd264496b73475fbb"} Jul 21 08:18:52.404868 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.404834 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-llppt" event={"ID":"a59731f0-2456-4447-a672-69a180edc896","Type":"ContainerStarted","Data":"eaf3d2285c8e40d2a087215ad636e7111f575210cfa6bf462a69e67a1795f110"} Jul 21 08:18:52.415753 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.415724 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal" event={"ID":"562d6f5337e483c37d48c2d1b5b30966","Type":"ContainerStarted","Data":"07cf612d8ab5f912ad83acdd2025ce4e842e9b4f8e041bb58de4f40e50433a1e"} Jul 21 08:18:52.422581 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.421950 2563 generic.go:356] "Generic (PLEG): container finished" podID="e7bcbd72e8164ffd2b41df6d71e8e46c" containerID="10663e280e945a99573dc93025bb84a24ed5a395fe1e035b2acbb084164c20ae" exitCode=0 Jul 21 08:18:52.422581 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.422196 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" event={"ID":"e7bcbd72e8164ffd2b41df6d71e8e46c","Type":"ContainerDied","Data":"10663e280e945a99573dc93025bb84a24ed5a395fe1e035b2acbb084164c20ae"} Jul 21 08:18:52.430403 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.430342 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-36.ec2.internal" podStartSLOduration=2.430323455 podStartE2EDuration="2.430323455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:18:52.42987148 +0000 UTC m=+4.508909818" watchObservedRunningTime="2026-07-21 08:18:52.430323455 +0000 UTC m=+4.509361791" Jul 21 08:18:52.438715 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.438629 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ptcvr" event={"ID":"d77c79ae-c248-4c1f-9a2b-b4b63460b83f","Type":"ContainerStarted","Data":"c3b62d157734bb38bc55d58970bda6437e4d28bb7332bf7f8b97f8ad509ca2b5"} Jul 21 08:18:52.441562 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.441482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" event={"ID":"25959877-ace6-4180-9de6-f2bd857a8ff7","Type":"ContainerStarted","Data":"74b2ed22ea5b482d3049f5238afcabecd16ab8d5073a1f11fe65e19903349ecb"} Jul 21 08:18:52.446592 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.446459 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5m66" event={"ID":"c044a2bf-898d-4e0c-b624-01d347f5d5f9","Type":"ContainerStarted","Data":"e1d9cc4c2a71b0a5eada9fff4e75e0a1ede8f815caaf4b30870c446db0a4e3fd"} Jul 21 08:18:52.451316 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:52.451289 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" event={"ID":"988708bc-ebff-4e44-9fd5-fde5f19594ed","Type":"ContainerStarted","Data":"92a02ff071cc292d3e17f6fc9a0c8aa2a6a8b2870bf260e5bebe3b7d0622c9a9"} Jul 21 08:18:53.489352 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:53.488789 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" event={"ID":"e7bcbd72e8164ffd2b41df6d71e8e46c","Type":"ContainerStarted","Data":"2a4878dd69bbdd4b6213cf20d495c8504d60453ac09afcb929b28441a0759750"} Jul 21 08:18:53.504226 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:53.504162 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-36.ec2.internal" podStartSLOduration=3.504138014 podStartE2EDuration="3.504138014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:18:53.503168898 +0000 UTC m=+5.582207234" watchObservedRunningTime="2026-07-21 08:18:53.504138014 +0000 UTC m=+5.583176348" Jul 21 08:18:53.899119 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:53.899028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:53.899295 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:53.899194 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:53.899295 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:53.899262 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret podName:ae3fb7e3-9363-4e30-b5c4-362303bd14bb nodeName:}" failed. No retries permitted until 2026-07-21 08:18:57.899241202 +0000 UTC m=+9.978279516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret") pod "global-pull-secret-syncer-fkhp9" (UID: "ae3fb7e3-9363-4e30-b5c4-362303bd14bb") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:53.999962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:53.999916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:54.000160 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:54.000128 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:54.000242 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:54.000207 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:18:58.000182539 +0000 UTC m=+10.079220892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:54.100721 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:54.100680 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:54.100929 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:54.100903 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:18:54.101028 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:54.100931 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:18:54.101028 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:54.100944 2563 projected.go:196] Error preparing data for projected volume kube-api-access-x42cd for pod openshift-network-diagnostics/network-check-target-88msb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:54.101028 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:54.101018 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd podName:131059bc-6588-40c7-a8c5-73664c7e59e4 nodeName:}" failed. No retries permitted until 2026-07-21 08:18:58.100998933 +0000 UTC m=+10.180037247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42cd" (UniqueName: "kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd") pod "network-check-target-88msb" (UID: "131059bc-6588-40c7-a8c5-73664c7e59e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:54.361787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:54.357732 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:54.361787 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:54.357927 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:18:54.361787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:54.358286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:54.361787 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:54.358588 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:18:54.361787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:54.358660 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:54.361787 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:54.358799 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:18:56.357245 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:56.356454 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:56.357245 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:56.356594 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:18:56.357245 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:56.356996 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:56.357245 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:56.357086 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:18:56.357245 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:56.357136 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:56.357245 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:56.357204 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:18:57.937272 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:57.937228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:57.937692 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:57.937400 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:57.937692 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:57.937477 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret podName:ae3fb7e3-9363-4e30-b5c4-362303bd14bb nodeName:}" failed. No retries permitted until 2026-07-21 08:19:05.937458129 +0000 UTC m=+18.016496444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret") pod "global-pull-secret-syncer-fkhp9" (UID: "ae3fb7e3-9363-4e30-b5c4-362303bd14bb") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:18:58.038038 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:58.038005 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:58.038194 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:58.038154 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:58.038263 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:58.038224 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:19:06.03820459 +0000 UTC m=+18.117242922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:18:58.139549 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:58.138847 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:58.139549 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:58.139060 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:18:58.139549 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:58.139081 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:18:58.139549 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:58.139093 2563 projected.go:196] Error preparing data for projected volume kube-api-access-x42cd for pod openshift-network-diagnostics/network-check-target-88msb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:58.139549 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:58.139156 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd podName:131059bc-6588-40c7-a8c5-73664c7e59e4 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:06.139135978 +0000 UTC m=+18.218174301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42cd" (UniqueName: "kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd") pod "network-check-target-88msb" (UID: "131059bc-6588-40c7-a8c5-73664c7e59e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:18:58.357104 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:58.356522 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:18:58.357104 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:58.356525 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:18:58.357104 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:58.356676 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:18:58.357104 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:58.356744 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:18:58.357104 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:18:58.356538 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:18:58.357104 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:18:58.356830 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:00.352144 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:00.352110 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:00.352144 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:00.352150 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:00.352887 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:00.352220 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:00.352887 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:00.352356 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:00.352887 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:00.352110 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:00.352887 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:00.352488 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:02.352088 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:02.352058 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:02.352679 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:02.352058 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:02.352679 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:02.352178 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:02.352679 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:02.352068 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:02.352679 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:02.352236 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:02.352679 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:02.352340 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:04.352342 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:04.352304 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:04.352879 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:04.352397 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:04.352879 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:04.352406 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:04.352879 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:04.352515 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:04.352879 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:04.352574 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:04.352879 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:04.352647 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:05.998517 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:05.998480 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:05.999013 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:05.998626 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:05.999013 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:05.998706 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret podName:ae3fb7e3-9363-4e30-b5c4-362303bd14bb nodeName:}" failed. No retries permitted until 2026-07-21 08:19:21.998683139 +0000 UTC m=+34.077721451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret") pod "global-pull-secret-syncer-fkhp9" (UID: "ae3fb7e3-9363-4e30-b5c4-362303bd14bb") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:06.099899 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:06.099839 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:06.100062 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:06.100009 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:06.100113 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:06.100073 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:19:22.100053739 +0000 UTC m=+34.179092070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:06.200750 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:06.200703 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:06.200921 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:06.200891 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:19:06.200921 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:06.200919 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:19:06.201044 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:06.200930 2563 projected.go:196] Error preparing data for projected volume kube-api-access-x42cd for pod openshift-network-diagnostics/network-check-target-88msb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:06.201044 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:06.200990 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd podName:131059bc-6588-40c7-a8c5-73664c7e59e4 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:22.200971174 +0000 UTC m=+34.280009502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42cd" (UniqueName: "kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd") pod "network-check-target-88msb" (UID: "131059bc-6588-40c7-a8c5-73664c7e59e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:06.352696 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:06.352611 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:06.352696 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:06.352636 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:06.352905 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:06.352611 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:06.352905 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:06.352744 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:06.352905 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:06.352840 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:06.353013 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:06.352935 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:08.352886 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:08.352850 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:08.353433 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:08.352915 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:08.353538 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:08.352952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:08.353538 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:08.353472 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:08.353636 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:08.353537 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:08.353636 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:08.353592 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:10.352068 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:10.352032 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:10.352068 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:10.352060 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:10.352727 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:10.352035 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:10.352727 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:10.352166 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:10.352727 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:10.352269 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:10.352727 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:10.352366 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:11.526367 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.525836 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ptcvr" event={"ID":"d77c79ae-c248-4c1f-9a2b-b4b63460b83f","Type":"ContainerStarted","Data":"71348728ba489aa8eaea73245696e05fd35c1c8f75515b4074e89afd2f3df8e4"} Jul 21 08:19:11.527510 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.527490 2563 generic.go:356] "Generic (PLEG): container finished" podID="25959877-ace6-4180-9de6-f2bd857a8ff7" containerID="f5db5c07317616468293c9917a213ef3cbc70a8fc05f0c26890ef83ccd38a12d" exitCode=0 Jul 21 08:19:11.527593 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.527570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" event={"ID":"25959877-ace6-4180-9de6-f2bd857a8ff7","Type":"ContainerDied","Data":"f5db5c07317616468293c9917a213ef3cbc70a8fc05f0c26890ef83ccd38a12d"} Jul 21 08:19:11.528829 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.528760 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5m66" event={"ID":"c044a2bf-898d-4e0c-b624-01d347f5d5f9","Type":"ContainerStarted","Data":"73b813e149abe94f70626e69b2d204d3a2fc123c02e7863ffa3bfbb0ed57e0d4"} Jul 21 08:19:11.530190 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.530169 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" event={"ID":"988708bc-ebff-4e44-9fd5-fde5f19594ed","Type":"ContainerStarted","Data":"a256d1ac54c50b9bafc8a79558107b50bbf5e445faf1a39cd1641456a690c911"} Jul 21 08:19:11.531635 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.531616 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" event={"ID":"e55e17f4-78a0-4e21-ab12-0ce0471ec960","Type":"ContainerStarted","Data":"dbb5ce3d4dbda97688ee5a821ffc51ee45b86d92cfedad0ae3793567221547f1"} Jul 21 08:19:11.533746 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.533700 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" event={"ID":"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d","Type":"ContainerStarted","Data":"0ae89a306ea6c282192e001490022eefa7d3c76ea26f10a8752d309424d7866b"} Jul 21 08:19:11.533813 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.533762 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" event={"ID":"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d","Type":"ContainerStarted","Data":"c59ae5ffe61bbd0dbc0bb3d2c503a265732edce35ca613e9c60348d3f18f8ca3"} Jul 21 08:19:11.533863 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.533811 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" event={"ID":"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d","Type":"ContainerStarted","Data":"e47f75c5e40edee479cc62fe0e67bf0ed52e5272976ed5be8171dda1e5209e1e"} Jul 21 08:19:11.533906 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.533891 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" event={"ID":"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d","Type":"ContainerStarted","Data":"4d5b26cbb79eb2f37b3b60c3f71a533a8b7966cd3238002d8602b416a9091d6b"} Jul 21 08:19:11.535253 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.535229 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5l9v" event={"ID":"39eb937b-c6b2-47bf-97ab-7ccd168023ec","Type":"ContainerStarted","Data":"0f67bf44dbe80b79a32e9c5921ef4a1423c5c0f2ed25e5abc06735f686df938a"} Jul 21 08:19:11.536519 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.536497 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-llppt" event={"ID":"a59731f0-2456-4447-a672-69a180edc896","Type":"ContainerStarted","Data":"98b63dabb98da80885233ad397dfb0a08387b5f16158a2233301dfad1d9ffc73"} Jul 21 08:19:11.544636 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.544599 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ptcvr" podStartSLOduration=3.030415129 podStartE2EDuration="22.544587554s" totalImagesPullingTime="19.514172425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:11.544129323 +0000 UTC m=+23.623167654" watchObservedRunningTime="2026-07-21 08:19:11.544587554 +0000 UTC m=+23.623625887" Jul 21 08:19:11.557064 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.557025 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m5l9v" podStartSLOduration=3.088123702 podStartE2EDuration="22.557012671s" totalImagesPullingTime="19.468888969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:11.556851372 +0000 UTC m=+23.635889703" watchObservedRunningTime="2026-07-21 08:19:11.557012671 +0000 UTC m=+23.636050999" Jul 21 08:19:11.588815 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.588758 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h5m66" podStartSLOduration=3.123620049 podStartE2EDuration="22.588743995s" totalImagesPullingTime="19.465123946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:11.588079919 +0000 UTC m=+23.667118251" watchObservedRunningTime="2026-07-21 08:19:11.588743995 +0000 UTC m=+23.667782327" Jul 21 08:19:11.605309 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.605262 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bwlz9" podStartSLOduration=3.137589824 podStartE2EDuration="22.605249851s" totalImagesPullingTime="19.467660027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:11.604805644 +0000 UTC m=+23.683843978" watchObservedRunningTime="2026-07-21 08:19:11.605249851 +0000 UTC m=+23.684288181" Jul 21 08:19:11.619190 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:11.619144 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-llppt" podStartSLOduration=3.148259351 podStartE2EDuration="22.61912694s" totalImagesPullingTime="19.470867589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:11.618431962 +0000 UTC m=+23.697470292" watchObservedRunningTime="2026-07-21 08:19:11.61912694 +0000 UTC m=+23.698165272" Jul 21 08:19:12.352361 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:12.352327 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:12.352554 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:12.352327 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:12.352554 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:12.352477 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:12.352554 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:12.352534 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:12.352698 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:12.352327 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:12.352698 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:12.352661 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:12.544058 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:12.543539 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9n8qb" event={"ID":"3c68db5d-94fa-4001-ad5a-985f8f568fce","Type":"ContainerStarted","Data":"a15d02dea3429f5ebaa98c950ffb40be48a211b28c3d353474eed54aff61fab8"} Jul 21 08:19:12.548110 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:12.548080 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" event={"ID":"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d","Type":"ContainerStarted","Data":"1c6fc042804adbd3107427796d7132c8dfecb3c51b1a8bd79c49bf62f0d4711a"} Jul 21 08:19:12.548247 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:12.548160 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" event={"ID":"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d","Type":"ContainerStarted","Data":"a2ab48ae42b7569a649e28838e0fbfacbd1e6d74129b927d1f6c71f11e1f95c7"} Jul 21 08:19:12.558135 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:12.558061 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9n8qb" podStartSLOduration=4.092640258 podStartE2EDuration="23.558040504s" totalImagesPullingTime="19.465400246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:12.557056397 +0000 UTC m=+24.636094740" watchObservedRunningTime="2026-07-21 08:19:12.558040504 +0000 UTC m=+24.637078872" Jul 21 08:19:12.634682 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:12.634658 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 08:19:13.314839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:13.314740 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T08:19:12.634677785Z","UUID":"8dc69346-a3ec-49fc-8d6c-3f0dfc229f29","Handler":null,"Name":"","Endpoint":""} Jul 21 08:19:13.316584 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:13.316558 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 08:19:13.316738 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:13.316601 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 08:19:13.551444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:13.551413 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" event={"ID":"e55e17f4-78a0-4e21-ab12-0ce0471ec960","Type":"ContainerStarted","Data":"5668bd9feb5e6a00ff1c79634d7ed37cd1bf44d6f5373c8d8e0e97929b944a07"} Jul 21 08:19:14.160456 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.160428 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:19:14.160978 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.160957 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:19:14.177006 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.176911 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:19:14.177488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.177468 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m5l9v" Jul 21 08:19:14.352230 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.352197 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:14.352423 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.352201 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:14.352423 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.352204 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:14.352423 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:14.352341 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:14.352563 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:14.352457 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:14.352563 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:14.352548 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:14.555489 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.555408 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" event={"ID":"e55e17f4-78a0-4e21-ab12-0ce0471ec960","Type":"ContainerStarted","Data":"d21125b4d15a2898f94a9c5ba9f42985cd07ad95db7819492b1e30c42b0dca04"} Jul 21 08:19:14.558579 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.558546 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" event={"ID":"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d","Type":"ContainerStarted","Data":"0988aec498b97c93275fee192cbda0a4632535c6ec9fb8bf9d7c90b7f8b044b3"} Jul 21 08:19:14.571055 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:14.570997 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gl8ms" podStartSLOduration=3.352101508 podStartE2EDuration="25.570979699s" totalImagesPullingTime="22.218878191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:14.57073562 +0000 UTC m=+26.649773953" watchObservedRunningTime="2026-07-21 08:19:14.570979699 +0000 UTC m=+26.650018029" Jul 21 08:19:16.352218 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.352056 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:16.352778 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.352061 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:16.352778 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:16.352301 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:16.352778 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.352118 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:16.352778 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:16.352367 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:16.353031 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:16.352840 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:16.568242 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.568082 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" event={"ID":"9b9bec6d-5a3a-45a7-be1a-e5aa116ce18d","Type":"ContainerStarted","Data":"cf6f41bbe8b59e6c4c26a68d76b2afc848ad357db5078f632de370cfd98cb376"} Jul 21 08:19:16.568419 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.568378 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:19:16.568518 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.568490 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:19:16.568573 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.568542 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:19:16.584080 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.584052 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:19:16.584575 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.584559 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:19:16.592887 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:16.592849 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" podStartSLOduration=8.038529848 podStartE2EDuration="27.592834883s" totalImagesPullingTime="19.554305035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:16.591581424 +0000 UTC m=+28.670619778" watchObservedRunningTime="2026-07-21 08:19:16.592834883 +0000 UTC m=+28.671873214" Jul 21 08:19:17.572552 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:17.572513 2563 generic.go:356] "Generic (PLEG): container finished" podID="25959877-ace6-4180-9de6-f2bd857a8ff7" containerID="5fd6dd963f76ee672b090ef4053bd055d705298096a9537556320d8501260028" exitCode=0 Jul 21 08:19:17.573182 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:17.572611 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" event={"ID":"25959877-ace6-4180-9de6-f2bd857a8ff7","Type":"ContainerDied","Data":"5fd6dd963f76ee672b090ef4053bd055d705298096a9537556320d8501260028"} Jul 21 08:19:18.305024 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:18.303432 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fkhp9"] Jul 21 08:19:18.305024 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:18.303606 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:18.305024 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:18.303734 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:18.307800 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:18.307770 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-88msb"] Jul 21 08:19:18.307939 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:18.307883 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:18.308001 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:18.307971 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:18.308517 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:18.308496 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7dgc9"] Jul 21 08:19:18.308630 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:18.308616 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:18.308746 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:18.308723 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:19.579550 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:19.579515 2563 generic.go:356] "Generic (PLEG): container finished" podID="25959877-ace6-4180-9de6-f2bd857a8ff7" containerID="a14a7e5240dd4cb66b6381385e2506793c8f37e238e4c956f39af1d0a7b70412" exitCode=0 Jul 21 08:19:19.580006 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:19.579559 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" event={"ID":"25959877-ace6-4180-9de6-f2bd857a8ff7","Type":"ContainerDied","Data":"a14a7e5240dd4cb66b6381385e2506793c8f37e238e4c956f39af1d0a7b70412"} Jul 21 08:19:20.352665 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:20.352599 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:20.352665 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:20.352616 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:20.352665 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:20.352601 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:20.352911 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:20.352719 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:20.352911 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:20.352781 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:20.352911 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:20.352888 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:21.591152 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:21.590918 2563 generic.go:356] "Generic (PLEG): container finished" podID="25959877-ace6-4180-9de6-f2bd857a8ff7" containerID="53cbc4afab08602eddeeb29d7792a62b7cfcdd24ee60cd1cf1e4f0c9d97dcc66" exitCode=0 Jul 21 08:19:21.591556 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:21.591018 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" event={"ID":"25959877-ace6-4180-9de6-f2bd857a8ff7","Type":"ContainerDied","Data":"53cbc4afab08602eddeeb29d7792a62b7cfcdd24ee60cd1cf1e4f0c9d97dcc66"} Jul 21 08:19:22.021016 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:22.020934 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:22.021155 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.021089 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:22.021192 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.021156 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret podName:ae3fb7e3-9363-4e30-b5c4-362303bd14bb nodeName:}" failed. No retries permitted until 2026-07-21 08:19:54.021137308 +0000 UTC m=+66.100175617 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret") pod "global-pull-secret-syncer-fkhp9" (UID: "ae3fb7e3-9363-4e30-b5c4-362303bd14bb") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:22.121764 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:22.121732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:22.121999 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.121850 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:22.122087 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.122060 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:19:54.122037182 +0000 UTC m=+66.201075516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:22.222651 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:22.222619 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:22.222831 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.222757 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:19:22.222831 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.222785 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:19:22.222831 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.222798 2563 projected.go:196] Error preparing data for projected volume kube-api-access-x42cd for pod openshift-network-diagnostics/network-check-target-88msb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:22.222993 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.222865 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd podName:131059bc-6588-40c7-a8c5-73664c7e59e4 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:54.222846845 +0000 UTC m=+66.301885182 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42cd" (UniqueName: "kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd") pod "network-check-target-88msb" (UID: "131059bc-6588-40c7-a8c5-73664c7e59e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:22.352179 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:22.352097 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:22.352179 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:22.352131 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:22.352349 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:22.352097 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:22.352349 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.352215 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88msb" podUID="131059bc-6588-40c7-a8c5-73664c7e59e4" Jul 21 08:19:22.352349 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.352288 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:19:22.352468 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:22.352376 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhp9" podUID="ae3fb7e3-9363-4e30-b5c4-362303bd14bb" Jul 21 08:19:23.777498 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.777473 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-36.ec2.internal" event="NodeReady" Jul 21 08:19:23.777912 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.777596 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 08:19:23.808929 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.808833 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv"] Jul 21 08:19:23.811229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.810703 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:23.815888 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.813337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 21 08:19:23.815888 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.813536 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rb9mm\"" Jul 21 08:19:23.815888 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.813552 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75676588dc-89tmc"] Jul 21 08:19:23.815888 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.813566 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 21 08:19:23.816603 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.816124 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:23.818479 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.818457 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 08:19:23.818665 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.818647 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 08:19:23.818754 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.818720 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 08:19:23.819368 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.819290 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hvc2x\"" Jul 21 08:19:23.822530 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.822172 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv"] Jul 21 08:19:23.824457 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.824435 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xws6r"] Jul 21 08:19:23.824993 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.824973 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 08:19:23.826815 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.826670 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:23.829242 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.828122 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75676588dc-89tmc"] Jul 21 08:19:23.829242 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.828191 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k8kwc"] Jul 21 08:19:23.829969 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.829932 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 08:19:23.830342 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.830122 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 08:19:23.830514 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.830494 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jw7kl\"" Jul 21 08:19:23.830616 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.830595 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8kwc" Jul 21 08:19:23.832734 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.832716 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 08:19:23.832995 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.832976 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 08:19:23.833130 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.833011 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xmzts\"" Jul 21 08:19:23.833236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.833033 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 08:19:23.841149 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.841070 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xws6r"] Jul 21 08:19:23.841149 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.841122 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8kwc"] Jul 21 08:19:23.937287 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937248 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-certificates\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:23.937488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbdz\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-kube-api-access-kzbdz\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:23.937488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937325 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/041c75cc-9740-48f5-8f2b-12043420de1e-cert\") pod \"ingress-canary-k8kwc\" (UID: \"041c75cc-9740-48f5-8f2b-12043420de1e\") " pod="openshift-ingress-canary/ingress-canary-k8kwc" Jul 21 08:19:23.937488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqvt4\" (UniqueName: \"kubernetes.io/projected/041c75cc-9740-48f5-8f2b-12043420de1e-kube-api-access-cqvt4\") pod \"ingress-canary-k8kwc\" (UID: \"041c75cc-9740-48f5-8f2b-12043420de1e\") " pod="openshift-ingress-canary/ingress-canary-k8kwc" Jul 21 08:19:23.937488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:23.937693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937516 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f8bcd643-8b1e-4672-a68b-759c36bfad9d-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:23.937693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937534 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:23.937693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937556 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:23.937693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937572 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-bound-sa-token\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:23.937693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937587 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f822e8b-5f1a-4335-8b98-928c1fb36ced-tmp-dir\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:23.937693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937610 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-installation-pull-secrets\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:23.937693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937654 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ps48\" (UniqueName: \"kubernetes.io/projected/0f822e8b-5f1a-4335-8b98-928c1fb36ced-kube-api-access-2ps48\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:23.937693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937678 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-trusted-ca\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:23.938018 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937706 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f822e8b-5f1a-4335-8b98-928c1fb36ced-config-volume\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:23.938018 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937799 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83c926ee-ff20-44a2-84da-31998e65e2fa-ca-trust-extracted\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:23.938018 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.937836 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-image-registry-private-configuration\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:23.952025 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:23.951995 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:24.033250 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.033214 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:24.039136 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039108 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ps48\" (UniqueName: \"kubernetes.io/projected/0f822e8b-5f1a-4335-8b98-928c1fb36ced-kube-api-access-2ps48\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:24.039253 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-trusted-ca\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.039253 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f822e8b-5f1a-4335-8b98-928c1fb36ced-config-volume\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:24.039253 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039200 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83c926ee-ff20-44a2-84da-31998e65e2fa-ca-trust-extracted\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.039253 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039234 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-image-registry-private-configuration\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.039469 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-certificates\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.039469 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbdz\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-kube-api-access-kzbdz\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.039469 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039330 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/041c75cc-9740-48f5-8f2b-12043420de1e-cert\") pod \"ingress-canary-k8kwc\" (UID: \"041c75cc-9740-48f5-8f2b-12043420de1e\") " pod="openshift-ingress-canary/ingress-canary-k8kwc" Jul 21 08:19:24.039469 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039350 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqvt4\" (UniqueName: \"kubernetes.io/projected/041c75cc-9740-48f5-8f2b-12043420de1e-kube-api-access-cqvt4\") pod \"ingress-canary-k8kwc\" (UID: \"041c75cc-9740-48f5-8f2b-12043420de1e\") " pod="openshift-ingress-canary/ingress-canary-k8kwc" Jul 21 08:19:24.039469 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:24.039469 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039449 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f8bcd643-8b1e-4672-a68b-759c36bfad9d-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:24.039754 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039472 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.039754 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.039577 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 08:19:24.039754 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.039591 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75676588dc-89tmc: secret "image-registry-tls" not found Jul 21 08:19:24.039754 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.039654 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls podName:83c926ee-ff20-44a2-84da-31998e65e2fa nodeName:}" failed. No retries permitted until 2026-07-21 08:19:24.539636276 +0000 UTC m=+36.618674586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls") pod "image-registry-75676588dc-89tmc" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa") : secret "image-registry-tls" not found Jul 21 08:19:24.039754 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039738 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83c926ee-ff20-44a2-84da-31998e65e2fa-ca-trust-extracted\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.040011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.039822 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f822e8b-5f1a-4335-8b98-928c1fb36ced-config-volume\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:24.040011 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.039841 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:24.040011 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.039891 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls podName:0f822e8b-5f1a-4335-8b98-928c1fb36ced nodeName:}" failed. No retries permitted until 2026-07-21 08:19:24.539873794 +0000 UTC m=+36.618912123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls") pod "dns-default-xws6r" (UID: "0f822e8b-5f1a-4335-8b98-928c1fb36ced") : secret "dns-default-metrics-tls" not found Jul 21 08:19:24.040155 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.040116 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:24.040214 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.040154 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-bound-sa-token\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.040214 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.040183 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f822e8b-5f1a-4335-8b98-928c1fb36ced-tmp-dir\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:24.040306 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.040210 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-installation-pull-secrets\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.040306 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.040255 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 08:19:24.040306 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.040295 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-trusted-ca\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.040476 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.040302 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert podName:f8bcd643-8b1e-4672-a68b-759c36bfad9d nodeName:}" failed. No retries permitted until 2026-07-21 08:19:24.540287677 +0000 UTC m=+36.619325987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-kwwxv" (UID: "f8bcd643-8b1e-4672-a68b-759c36bfad9d") : secret "networking-console-plugin-cert" not found Jul 21 08:19:24.040476 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.040336 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-certificates\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.040476 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.040407 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f8bcd643-8b1e-4672-a68b-759c36bfad9d-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:24.040635 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.040570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f822e8b-5f1a-4335-8b98-928c1fb36ced-tmp-dir\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:24.044330 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.044307 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-image-registry-private-configuration\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.044330 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.044311 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/041c75cc-9740-48f5-8f2b-12043420de1e-cert\") pod \"ingress-canary-k8kwc\" (UID: \"041c75cc-9740-48f5-8f2b-12043420de1e\") " pod="openshift-ingress-canary/ingress-canary-k8kwc" Jul 21 08:19:24.044499 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.044337 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-installation-pull-secrets\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.050304 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.050275 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbdz\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-kube-api-access-kzbdz\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.050562 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.050544 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqvt4\" (UniqueName: \"kubernetes.io/projected/041c75cc-9740-48f5-8f2b-12043420de1e-kube-api-access-cqvt4\") pod \"ingress-canary-k8kwc\" (UID: \"041c75cc-9740-48f5-8f2b-12043420de1e\") " pod="openshift-ingress-canary/ingress-canary-k8kwc" Jul 21 08:19:24.050623 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.050585 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ps48\" (UniqueName: \"kubernetes.io/projected/0f822e8b-5f1a-4335-8b98-928c1fb36ced-kube-api-access-2ps48\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:24.050937 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.050902 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-bound-sa-token\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.133613 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.133534 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:24.148816 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.148790 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8kwc" Jul 21 08:19:24.293602 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.293574 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8kwc"] Jul 21 08:19:24.297718 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:19:24.297691 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041c75cc_9740_48f5_8f2b_12043420de1e.slice/crio-00f1bb73e7b0734545d58305597eca53297e824ffba1493c74194cab2d7f1131 WatchSource:0}: Error finding container 00f1bb73e7b0734545d58305597eca53297e824ffba1493c74194cab2d7f1131: Status 404 returned error can't find the container with id 00f1bb73e7b0734545d58305597eca53297e824ffba1493c74194cab2d7f1131 Jul 21 08:19:24.352479 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.352439 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:24.352666 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.352439 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:24.352666 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.352439 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:24.355822 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.355769 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 08:19:24.355967 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.355876 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 08:19:24.355967 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.355928 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 08:19:24.356236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.356215 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 08:19:24.356338 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.356282 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4qzvw\"" Jul 21 08:19:24.356506 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.356489 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bh656\"" Jul 21 08:19:24.544394 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.544329 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:24.544580 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.544447 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:24.544580 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.544492 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:24.544726 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.544587 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 08:19:24.544726 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.544590 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 08:19:24.544726 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.544601 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:24.544726 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.544665 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert podName:f8bcd643-8b1e-4672-a68b-759c36bfad9d nodeName:}" failed. No retries permitted until 2026-07-21 08:19:25.544645992 +0000 UTC m=+37.623684304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-kwwxv" (UID: "f8bcd643-8b1e-4672-a68b-759c36bfad9d") : secret "networking-console-plugin-cert" not found Jul 21 08:19:24.544726 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.544607 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75676588dc-89tmc: secret "image-registry-tls" not found Jul 21 08:19:24.544726 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.544696 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls podName:0f822e8b-5f1a-4335-8b98-928c1fb36ced nodeName:}" failed. No retries permitted until 2026-07-21 08:19:25.544677592 +0000 UTC m=+37.623715907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls") pod "dns-default-xws6r" (UID: "0f822e8b-5f1a-4335-8b98-928c1fb36ced") : secret "dns-default-metrics-tls" not found Jul 21 08:19:24.544726 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:24.544729 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls podName:83c926ee-ff20-44a2-84da-31998e65e2fa nodeName:}" failed. No retries permitted until 2026-07-21 08:19:25.54471819 +0000 UTC m=+37.623756500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls") pod "image-registry-75676588dc-89tmc" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa") : secret "image-registry-tls" not found Jul 21 08:19:24.600311 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:24.600281 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8kwc" event={"ID":"041c75cc-9740-48f5-8f2b-12043420de1e","Type":"ContainerStarted","Data":"00f1bb73e7b0734545d58305597eca53297e824ffba1493c74194cab2d7f1131"} Jul 21 08:19:25.554396 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:25.554115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:25.555098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:25.554443 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:25.555098 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:25.554276 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:25.555098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:25.554484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:25.555098 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:25.554565 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls podName:0f822e8b-5f1a-4335-8b98-928c1fb36ced nodeName:}" failed. No retries permitted until 2026-07-21 08:19:27.554538654 +0000 UTC m=+39.633576980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls") pod "dns-default-xws6r" (UID: "0f822e8b-5f1a-4335-8b98-928c1fb36ced") : secret "dns-default-metrics-tls" not found Jul 21 08:19:25.555098 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:25.554583 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 08:19:25.555098 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:25.554588 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 08:19:25.555098 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:25.554601 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75676588dc-89tmc: secret "image-registry-tls" not found Jul 21 08:19:25.555098 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:25.554652 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert podName:f8bcd643-8b1e-4672-a68b-759c36bfad9d nodeName:}" failed. No retries permitted until 2026-07-21 08:19:27.554636419 +0000 UTC m=+39.633674756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-kwwxv" (UID: "f8bcd643-8b1e-4672-a68b-759c36bfad9d") : secret "networking-console-plugin-cert" not found Jul 21 08:19:25.555098 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:25.554670 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls podName:83c926ee-ff20-44a2-84da-31998e65e2fa nodeName:}" failed. No retries permitted until 2026-07-21 08:19:27.554660956 +0000 UTC m=+39.633699272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls") pod "image-registry-75676588dc-89tmc" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa") : secret "image-registry-tls" not found Jul 21 08:19:26.974943 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:26.974903 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:27.042864 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:27.042834 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:27.571437 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:27.571394 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:27.571598 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:27.571448 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:27.571598 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:27.571489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:27.571598 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:27.571543 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:27.571717 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:27.571609 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls podName:0f822e8b-5f1a-4335-8b98-928c1fb36ced nodeName:}" failed. No retries permitted until 2026-07-21 08:19:31.571587726 +0000 UTC m=+43.650626034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls") pod "dns-default-xws6r" (UID: "0f822e8b-5f1a-4335-8b98-928c1fb36ced") : secret "dns-default-metrics-tls" not found Jul 21 08:19:27.571717 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:27.571618 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 08:19:27.571717 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:27.571639 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75676588dc-89tmc: secret "image-registry-tls" not found Jul 21 08:19:27.571717 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:27.571623 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 08:19:27.571717 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:27.571678 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls podName:83c926ee-ff20-44a2-84da-31998e65e2fa nodeName:}" failed. No retries permitted until 2026-07-21 08:19:31.571666623 +0000 UTC m=+43.650704932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls") pod "image-registry-75676588dc-89tmc" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa") : secret "image-registry-tls" not found Jul 21 08:19:27.571870 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:27.571728 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert podName:f8bcd643-8b1e-4672-a68b-759c36bfad9d nodeName:}" failed. No retries permitted until 2026-07-21 08:19:31.5717142 +0000 UTC m=+43.650752521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-kwwxv" (UID: "f8bcd643-8b1e-4672-a68b-759c36bfad9d") : secret "networking-console-plugin-cert" not found Jul 21 08:19:27.611879 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:27.611845 2563 generic.go:356] "Generic (PLEG): container finished" podID="25959877-ace6-4180-9de6-f2bd857a8ff7" containerID="e5e9a63205a79d5de2c21064f1abc2ff35826c50d25889fa7ed9ef4e5de18b0b" exitCode=0 Jul 21 08:19:27.612037 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:27.611926 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" event={"ID":"25959877-ace6-4180-9de6-f2bd857a8ff7","Type":"ContainerDied","Data":"e5e9a63205a79d5de2c21064f1abc2ff35826c50d25889fa7ed9ef4e5de18b0b"} Jul 21 08:19:27.613531 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:27.613504 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8kwc" event={"ID":"041c75cc-9740-48f5-8f2b-12043420de1e","Type":"ContainerStarted","Data":"0e978bfdcb37c22b64b43f5f4e985251ab80fba549d31af622a07f52c61a4fed"} Jul 21 08:19:27.643823 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:27.643776 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k8kwc" podStartSLOduration=1.527003601 podStartE2EDuration="4.643757592s" totalImagesPullingTime="3.116753991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:27.643338351 +0000 UTC m=+39.722376682" watchObservedRunningTime="2026-07-21 08:19:27.643757592 +0000 UTC m=+39.722795926" Jul 21 08:19:28.619527 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:28.619285 2563 generic.go:356] "Generic (PLEG): container finished" podID="25959877-ace6-4180-9de6-f2bd857a8ff7" containerID="6e9002974b84ef2635dfcf0bb9696fff3414a51bda6d608a92270ddbaf012c34" exitCode=0 Jul 21 08:19:28.619929 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:28.619373 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" event={"ID":"25959877-ace6-4180-9de6-f2bd857a8ff7","Type":"ContainerDied","Data":"6e9002974b84ef2635dfcf0bb9696fff3414a51bda6d608a92270ddbaf012c34"} Jul 21 08:19:28.933011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:28.932977 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:29.035682 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:29.035653 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:29.292282 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:29.292204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:29.393107 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:29.393066 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:29.625317 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:29.625248 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" event={"ID":"25959877-ace6-4180-9de6-f2bd857a8ff7","Type":"ContainerStarted","Data":"51f10cd7c304317b0b4daf225b95bc27724f99e6ed7f8fb7cb59073197f90ec4"} Jul 21 08:19:29.645800 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:29.645758 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rvbx6" podStartSLOduration=8.470797509 podStartE2EDuration="40.645739736s" totalImagesPullingTime="32.174942227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:29.64453399 +0000 UTC m=+41.723572321" watchObservedRunningTime="2026-07-21 08:19:29.645739736 +0000 UTC m=+41.724778070" Jul 21 08:19:29.764076 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:29.764042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:29.825464 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:29.825433 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:31.348778 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:31.348748 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:31.449076 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:31.449034 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:31.602123 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:31.602031 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:31.602123 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:31.602086 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:31.602123 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:31.602113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:31.602331 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:31.602204 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:31.602331 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:31.602217 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 08:19:31.602331 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:31.602208 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 08:19:31.602331 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:31.602267 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls podName:0f822e8b-5f1a-4335-8b98-928c1fb36ced nodeName:}" failed. No retries permitted until 2026-07-21 08:19:39.602251733 +0000 UTC m=+51.681290043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls") pod "dns-default-xws6r" (UID: "0f822e8b-5f1a-4335-8b98-928c1fb36ced") : secret "dns-default-metrics-tls" not found Jul 21 08:19:31.602331 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:31.602273 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75676588dc-89tmc: secret "image-registry-tls" not found Jul 21 08:19:31.602331 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:31.602282 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert podName:f8bcd643-8b1e-4672-a68b-759c36bfad9d nodeName:}" failed. No retries permitted until 2026-07-21 08:19:39.602275976 +0000 UTC m=+51.681314284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-kwwxv" (UID: "f8bcd643-8b1e-4672-a68b-759c36bfad9d") : secret "networking-console-plugin-cert" not found Jul 21 08:19:31.602331 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:31.602300 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls podName:83c926ee-ff20-44a2-84da-31998e65e2fa nodeName:}" failed. No retries permitted until 2026-07-21 08:19:39.602287569 +0000 UTC m=+51.681325877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls") pod "image-registry-75676588dc-89tmc" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa") : secret "image-registry-tls" not found Jul 21 08:19:31.874014 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:31.873945 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:32.853256 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:32.853223 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:32.905705 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:32.905676 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:37.200657 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:37.200622 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:37.249378 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:37.249347 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:37.750317 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:37.750276 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:37.851313 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:37.851281 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:39.626908 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:39.626875 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:39.661605 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:39.661577 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:39.661751 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:39.661622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:39.661751 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:39.661657 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:39.661751 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:39.661733 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:39.661861 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:39.661781 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 08:19:39.661861 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:39.661798 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75676588dc-89tmc: secret "image-registry-tls" not found Jul 21 08:19:39.661861 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:39.661785 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls podName:0f822e8b-5f1a-4335-8b98-928c1fb36ced nodeName:}" failed. No retries permitted until 2026-07-21 08:19:55.661767949 +0000 UTC m=+67.740806273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls") pod "dns-default-xws6r" (UID: "0f822e8b-5f1a-4335-8b98-928c1fb36ced") : secret "dns-default-metrics-tls" not found Jul 21 08:19:39.661861 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:39.661856 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls podName:83c926ee-ff20-44a2-84da-31998e65e2fa nodeName:}" failed. No retries permitted until 2026-07-21 08:19:55.661843293 +0000 UTC m=+67.740881601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls") pod "image-registry-75676588dc-89tmc" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa") : secret "image-registry-tls" not found Jul 21 08:19:39.661861 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:39.661785 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 08:19:39.662030 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:39.661886 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert podName:f8bcd643-8b1e-4672-a68b-759c36bfad9d nodeName:}" failed. No retries permitted until 2026-07-21 08:19:55.661879694 +0000 UTC m=+67.740918003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-kwwxv" (UID: "f8bcd643-8b1e-4672-a68b-759c36bfad9d") : secret "networking-console-plugin-cert" not found Jul 21 08:19:39.728317 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:39.728289 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:44.006702 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:44.006669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:44.106640 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:44.106609 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:44.840674 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:44.840631 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:48.596225 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:48.596195 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6tbn" Jul 21 08:19:48.744828 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:48.744773 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:48.794070 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:48.794042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:19:54.074172 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.074129 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:54.076898 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.076880 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 08:19:54.088902 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.088874 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae3fb7e3-9363-4e30-b5c4-362303bd14bb-original-pull-secret\") pod \"global-pull-secret-syncer-fkhp9\" (UID: \"ae3fb7e3-9363-4e30-b5c4-362303bd14bb\") " pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:54.174900 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.174866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:19:54.177175 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.177147 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 08:19:54.185073 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:54.185040 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 08:19:54.185155 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:54.185143 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:20:58.185122591 +0000 UTC m=+130.264160901 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : secret "metrics-daemon-secret" not found Jul 21 08:19:54.275749 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.275701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:54.277929 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.277914 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 08:19:54.288578 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.288558 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 08:19:54.299978 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.299956 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42cd\" (UniqueName: \"kubernetes.io/projected/131059bc-6588-40c7-a8c5-73664c7e59e4-kube-api-access-x42cd\") pod \"network-check-target-88msb\" (UID: \"131059bc-6588-40c7-a8c5-73664c7e59e4\") " pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:54.365989 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.365918 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhp9" Jul 21 08:19:54.375176 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.375159 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4qzvw\"" Jul 21 08:19:54.383293 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.383275 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:19:54.494828 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.494794 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fkhp9"] Jul 21 08:19:54.498952 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:19:54.498926 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3fb7e3_9363_4e30_b5c4_362303bd14bb.slice/crio-c4df2ffe880a82b707bbdc9d0d5c97a1f4ae735cabd12a6c608013280a7dfb20 WatchSource:0}: Error finding container c4df2ffe880a82b707bbdc9d0d5c97a1f4ae735cabd12a6c608013280a7dfb20: Status 404 returned error can't find the container with id c4df2ffe880a82b707bbdc9d0d5c97a1f4ae735cabd12a6c608013280a7dfb20 Jul 21 08:19:54.510159 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.510136 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-88msb"] Jul 21 08:19:54.513084 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:19:54.513061 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131059bc_6588_40c7_a8c5_73664c7e59e4.slice/crio-b17f74f2161904c18b8fbfbeb60ba462fcaf38cef682acf0e005c9344ad32054 WatchSource:0}: Error finding container b17f74f2161904c18b8fbfbeb60ba462fcaf38cef682acf0e005c9344ad32054: Status 404 returned error can't find the container with id b17f74f2161904c18b8fbfbeb60ba462fcaf38cef682acf0e005c9344ad32054 Jul 21 08:19:54.684939 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.684904 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-88msb" event={"ID":"131059bc-6588-40c7-a8c5-73664c7e59e4","Type":"ContainerStarted","Data":"b17f74f2161904c18b8fbfbeb60ba462fcaf38cef682acf0e005c9344ad32054"} Jul 21 08:19:54.685891 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:54.685869 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fkhp9" event={"ID":"ae3fb7e3-9363-4e30-b5c4-362303bd14bb","Type":"ContainerStarted","Data":"c4df2ffe880a82b707bbdc9d0d5c97a1f4ae735cabd12a6c608013280a7dfb20"} Jul 21 08:19:55.687063 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:55.687025 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:19:55.687574 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:55.687083 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:19:55.687574 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:19:55.687114 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:19:55.687574 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:55.687260 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 08:19:55.687574 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:55.687262 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:55.687574 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:55.687320 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 08:19:55.687574 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:55.687331 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls podName:0f822e8b-5f1a-4335-8b98-928c1fb36ced nodeName:}" failed. No retries permitted until 2026-07-21 08:20:27.687309448 +0000 UTC m=+99.766347780 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls") pod "dns-default-xws6r" (UID: "0f822e8b-5f1a-4335-8b98-928c1fb36ced") : secret "dns-default-metrics-tls" not found Jul 21 08:19:55.687574 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:55.687274 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75676588dc-89tmc: secret "image-registry-tls" not found Jul 21 08:19:55.687574 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:55.687370 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert podName:f8bcd643-8b1e-4672-a68b-759c36bfad9d nodeName:}" failed. No retries permitted until 2026-07-21 08:20:27.687354225 +0000 UTC m=+99.766392551 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-kwwxv" (UID: "f8bcd643-8b1e-4672-a68b-759c36bfad9d") : secret "networking-console-plugin-cert" not found Jul 21 08:19:55.687574 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:19:55.687414 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls podName:83c926ee-ff20-44a2-84da-31998e65e2fa nodeName:}" failed. No retries permitted until 2026-07-21 08:20:27.687397185 +0000 UTC m=+99.766435507 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls") pod "image-registry-75676588dc-89tmc" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa") : secret "image-registry-tls" not found Jul 21 08:20:00.186780 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:00.186744 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:20:00.238049 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:00.238024 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:20:00.704515 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:00.704482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-88msb" event={"ID":"131059bc-6588-40c7-a8c5-73664c7e59e4","Type":"ContainerStarted","Data":"6369818f5e0b4dffcd08ea8821b61233d45b64065c82595f1db2975ac8e7b60a"} Jul 21 08:20:00.704709 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:00.704559 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:20:00.705916 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:00.705891 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fkhp9" event={"ID":"ae3fb7e3-9363-4e30-b5c4-362303bd14bb","Type":"ContainerStarted","Data":"783a353cc7cd0e54c4bb749c084f1b1fb427278f02fed25561e7d9f14bc0aa27"} Jul 21 08:20:00.718986 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:00.718942 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-88msb" podStartSLOduration=66.465078703 podStartE2EDuration="1m11.71892967s" totalImagesPullingTime="5.253850967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:20:00.718310444 +0000 UTC m=+72.797348776" watchObservedRunningTime="2026-07-21 08:20:00.71892967 +0000 UTC m=+72.797967998" Jul 21 08:20:00.731805 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:00.731760 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fkhp9" podStartSLOduration=66.460389621 podStartE2EDuration="1m11.731746872s" totalImagesPullingTime="5.271357251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:20:00.730695919 +0000 UTC m=+72.809734251" watchObservedRunningTime="2026-07-21 08:20:00.731746872 +0000 UTC m=+72.810785202" Jul 21 08:20:05.101757 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:05.101722 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:20:05.152864 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:05.152830 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:20:27.718705 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:27.718666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:20:27.719161 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:27.718731 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:20:27.719161 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:27.718771 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:20:27.719161 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:20:27.718827 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:20:27.719161 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:20:27.718868 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 08:20:27.719161 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:20:27.718884 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75676588dc-89tmc: secret "image-registry-tls" not found Jul 21 08:20:27.719161 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:20:27.718886 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 08:20:27.719161 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:20:27.718910 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls podName:0f822e8b-5f1a-4335-8b98-928c1fb36ced nodeName:}" failed. No retries permitted until 2026-07-21 08:21:31.718892795 +0000 UTC m=+163.797931107 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls") pod "dns-default-xws6r" (UID: "0f822e8b-5f1a-4335-8b98-928c1fb36ced") : secret "dns-default-metrics-tls" not found Jul 21 08:20:27.719161 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:20:27.718934 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls podName:83c926ee-ff20-44a2-84da-31998e65e2fa nodeName:}" failed. No retries permitted until 2026-07-21 08:21:31.718922302 +0000 UTC m=+163.797960611 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls") pod "image-registry-75676588dc-89tmc" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa") : secret "image-registry-tls" not found Jul 21 08:20:27.719161 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:20:27.718947 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert podName:f8bcd643-8b1e-4672-a68b-759c36bfad9d nodeName:}" failed. No retries permitted until 2026-07-21 08:21:31.718941248 +0000 UTC m=+163.797979556 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-kwwxv" (UID: "f8bcd643-8b1e-4672-a68b-759c36bfad9d") : secret "networking-console-plugin-cert" not found Jul 21 08:20:31.159115 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:31.159083 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:20:31.215303 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:31.215268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:20:31.710759 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:31.710727 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-88msb" Jul 21 08:20:58.228452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:20:58.228403 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:20:58.228936 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:20:58.228567 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 08:20:58.228936 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:20:58.228644 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:23:00.228628422 +0000 UTC m=+252.307666731 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : secret "metrics-daemon-secret" not found Jul 21 08:21:26.825655 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:26.825609 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" podUID="f8bcd643-8b1e-4672-a68b-759c36bfad9d" Jul 21 08:21:26.834235 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:26.834210 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-75676588dc-89tmc" podUID="83c926ee-ff20-44a2-84da-31998e65e2fa" Jul 21 08:21:26.842482 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:26.842461 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xws6r" podUID="0f822e8b-5f1a-4335-8b98-928c1fb36ced" Jul 21 08:21:26.904445 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:21:26.904424 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:21:26.904529 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:21:26.904424 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xws6r" Jul 21 08:21:26.904598 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:21:26.904424 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:21:27.377504 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:27.377458 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:21:31.772533 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:21:31.772492 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:21:31.772954 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:21:31.772543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:21:31.772954 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:31.772632 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 08:21:31.772954 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:31.772635 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:21:31.772954 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:21:31.772655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:21:31.772954 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:31.772642 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75676588dc-89tmc: secret "image-registry-tls" not found Jul 21 08:21:31.772954 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:31.772752 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 08:21:31.772954 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:31.772712 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls podName:0f822e8b-5f1a-4335-8b98-928c1fb36ced nodeName:}" failed. No retries permitted until 2026-07-21 08:23:33.772692025 +0000 UTC m=+285.851730338 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls") pod "dns-default-xws6r" (UID: "0f822e8b-5f1a-4335-8b98-928c1fb36ced") : secret "dns-default-metrics-tls" not found Jul 21 08:21:31.772954 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:31.772778 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls podName:83c926ee-ff20-44a2-84da-31998e65e2fa nodeName:}" failed. No retries permitted until 2026-07-21 08:23:33.772765005 +0000 UTC m=+285.851803321 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls") pod "image-registry-75676588dc-89tmc" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa") : secret "image-registry-tls" not found Jul 21 08:21:31.772954 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:21:31.772801 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert podName:f8bcd643-8b1e-4672-a68b-759c36bfad9d nodeName:}" failed. No retries permitted until 2026-07-21 08:23:33.772789646 +0000 UTC m=+285.851827954 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-kwwxv" (UID: "f8bcd643-8b1e-4672-a68b-759c36bfad9d") : secret "networking-console-plugin-cert" not found Jul 21 08:21:39.352461 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:21:39.352336 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:22:39.247160 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:22:39.247126 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:00.290636 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:00.290587 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:23:00.291096 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:00.290737 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 08:23:00.291096 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:00.290799 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs podName:19a14646-a2c6-4a63-8716-6fb7127293cf nodeName:}" failed. No retries permitted until 2026-07-21 08:25:02.290783162 +0000 UTC m=+374.369821471 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs") pod "network-metrics-daemon-7dgc9" (UID: "19a14646-a2c6-4a63-8716-6fb7127293cf") : secret "metrics-daemon-secret" not found Jul 21 08:23:06.942563 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.942531 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-nxz4z"] Jul 21 08:23:06.944464 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.944450 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:06.947470 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.947441 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw"] Jul 21 08:23:06.947808 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.947787 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 08:23:06.948762 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.948743 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 08:23:06.948854 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.948768 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 08:23:06.948854 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.948744 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lkzq8\"" Jul 21 08:23:06.949111 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.949094 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:06.957470 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.957402 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 08:23:06.957841 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.957816 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 08:23:06.957930 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.957827 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 08:23:06.958012 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.957990 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 08:23:06.958865 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.958846 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zxk5f\"" Jul 21 08:23:06.959026 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.959006 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69"] Jul 21 08:23:06.959077 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.959064 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 08:23:06.961311 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.961297 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:06.962996 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.962970 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-nxz4z"] Jul 21 08:23:06.963499 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.963481 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 08:23:06.963639 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.963622 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 21 08:23:06.963742 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.963725 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 21 08:23:06.963995 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.963982 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 21 08:23:06.964067 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.963989 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z56d2\"" Jul 21 08:23:06.971433 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.971413 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw"] Jul 21 08:23:06.974143 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:06.974123 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69"] Jul 21 08:23:07.036772 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.036734 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv"] Jul 21 08:23:07.038079 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038053 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk4bk\" (UniqueName: \"kubernetes.io/projected/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-kube-api-access-dk4bk\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:07.038171 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038089 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115156cb-6b04-420e-a460-b61a69906ef2-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.038171 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038107 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72316d11-75cc-4fa6-8a04-f21232313286-serving-cert\") pod \"service-ca-operator-5bcbc94c78-czhgw\" (UID: \"72316d11-75cc-4fa6-8a04-f21232313286\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.038171 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038152 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/115156cb-6b04-420e-a460-b61a69906ef2-snapshots\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.038273 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038198 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72316d11-75cc-4fa6-8a04-f21232313286-config\") pod \"service-ca-operator-5bcbc94c78-czhgw\" (UID: \"72316d11-75cc-4fa6-8a04-f21232313286\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.038273 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038240 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/115156cb-6b04-420e-a460-b61a69906ef2-tmp\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.038334 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038267 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/115156cb-6b04-420e-a460-b61a69906ef2-serving-cert\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.038334 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038300 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:07.038334 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038316 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4tt\" (UniqueName: \"kubernetes.io/projected/72316d11-75cc-4fa6-8a04-f21232313286-kube-api-access-9r4tt\") pod \"service-ca-operator-5bcbc94c78-czhgw\" (UID: \"72316d11-75cc-4fa6-8a04-f21232313286\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.038488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038333 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115156cb-6b04-420e-a460-b61a69906ef2-service-ca-bundle\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.038488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038417 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8xc\" (UniqueName: \"kubernetes.io/projected/115156cb-6b04-420e-a460-b61a69906ef2-kube-api-access-9t8xc\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.038760 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.038743 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.041119 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.041098 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8566ccfd7b-fdt46"] Jul 21 08:23:07.041636 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.041619 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 08:23:07.041905 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.041887 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 08:23:07.042111 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.042096 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lgqmv\"" Jul 21 08:23:07.042111 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.042107 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 08:23:07.042656 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.042643 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 08:23:07.043049 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.043034 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.046140 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.046124 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 21 08:23:07.046311 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.046298 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 21 08:23:07.046371 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.046331 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 21 08:23:07.046454 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.046401 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 21 08:23:07.046560 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.046547 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p6xsn\"" Jul 21 08:23:07.046693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.046681 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 08:23:07.046807 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.046797 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 08:23:07.063668 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.063643 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv"] Jul 21 08:23:07.066813 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.066792 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8566ccfd7b-fdt46"] Jul 21 08:23:07.139760 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/115156cb-6b04-420e-a460-b61a69906ef2-snapshots\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.139760 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139759 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72316d11-75cc-4fa6-8a04-f21232313286-config\") pod \"service-ca-operator-5bcbc94c78-czhgw\" (UID: \"72316d11-75cc-4fa6-8a04-f21232313286\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.139999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139778 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/115156cb-6b04-420e-a460-b61a69906ef2-tmp\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.139999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96x6r\" (UniqueName: \"kubernetes.io/projected/f28f0059-e25f-43ee-bab5-8cb570b14aae-kube-api-access-96x6r\") pod \"kube-storage-version-migrator-operator-7658d9b749-6zrvv\" (UID: \"f28f0059-e25f-43ee-bab5-8cb570b14aae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.139999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139836 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f28f0059-e25f-43ee-bab5-8cb570b14aae-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6zrvv\" (UID: \"f28f0059-e25f-43ee-bab5-8cb570b14aae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.139999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139880 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/115156cb-6b04-420e-a460-b61a69906ef2-serving-cert\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.139999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139906 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-stats-auth\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.139999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139933 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:07.139999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4tt\" (UniqueName: \"kubernetes.io/projected/72316d11-75cc-4fa6-8a04-f21232313286-kube-api-access-9r4tt\") pod \"service-ca-operator-5bcbc94c78-czhgw\" (UID: \"72316d11-75cc-4fa6-8a04-f21232313286\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.139999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.139967 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115156cb-6b04-420e-a460-b61a69906ef2-service-ca-bundle\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8xc\" (UniqueName: \"kubernetes.io/projected/115156cb-6b04-420e-a460-b61a69906ef2-kube-api-access-9t8xc\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppt7\" (UniqueName: \"kubernetes.io/projected/bae2b6a9-8844-42aa-9c78-1d931caed9d4-kube-api-access-4ppt7\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.140069 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140084 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk4bk\" (UniqueName: \"kubernetes.io/projected/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-kube-api-access-dk4bk\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115156cb-6b04-420e-a460-b61a69906ef2-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.140134 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls podName:7e3135af-6d9a-4fdc-86c5-604b3e7e25e2 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:07.640114447 +0000 UTC m=+259.719152761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-d7t69" (UID: "7e3135af-6d9a-4fdc-86c5-604b3e7e25e2") : secret "samples-operator-tls" not found Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72316d11-75cc-4fa6-8a04-f21232313286-serving-cert\") pod \"service-ca-operator-5bcbc94c78-czhgw\" (UID: \"72316d11-75cc-4fa6-8a04-f21232313286\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140185 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/115156cb-6b04-420e-a460-b61a69906ef2-tmp\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140256 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f28f0059-e25f-43ee-bab5-8cb570b14aae-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6zrvv\" (UID: \"f28f0059-e25f-43ee-bab5-8cb570b14aae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140279 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-default-certificate\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.140424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140417 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72316d11-75cc-4fa6-8a04-f21232313286-config\") pod \"service-ca-operator-5bcbc94c78-czhgw\" (UID: \"72316d11-75cc-4fa6-8a04-f21232313286\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.141011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140498 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/115156cb-6b04-420e-a460-b61a69906ef2-snapshots\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.141011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140873 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115156cb-6b04-420e-a460-b61a69906ef2-service-ca-bundle\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.141011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.140959 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115156cb-6b04-420e-a460-b61a69906ef2-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.142477 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.142450 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/115156cb-6b04-420e-a460-b61a69906ef2-serving-cert\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.142602 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.142564 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72316d11-75cc-4fa6-8a04-f21232313286-serving-cert\") pod \"service-ca-operator-5bcbc94c78-czhgw\" (UID: \"72316d11-75cc-4fa6-8a04-f21232313286\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.149704 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.149680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4tt\" (UniqueName: \"kubernetes.io/projected/72316d11-75cc-4fa6-8a04-f21232313286-kube-api-access-9r4tt\") pod \"service-ca-operator-5bcbc94c78-czhgw\" (UID: \"72316d11-75cc-4fa6-8a04-f21232313286\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.150128 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.150111 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8xc\" (UniqueName: \"kubernetes.io/projected/115156cb-6b04-420e-a460-b61a69906ef2-kube-api-access-9t8xc\") pod \"insights-operator-555d9d7885-nxz4z\" (UID: \"115156cb-6b04-420e-a460-b61a69906ef2\") " pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.150202 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.150147 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk4bk\" (UniqueName: \"kubernetes.io/projected/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-kube-api-access-dk4bk\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:07.225057 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.224980 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:07.241020 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.240998 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f28f0059-e25f-43ee-bab5-8cb570b14aae-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6zrvv\" (UID: \"f28f0059-e25f-43ee-bab5-8cb570b14aae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.241086 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.241024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-default-certificate\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.241086 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.241040 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.241086 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.241066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96x6r\" (UniqueName: \"kubernetes.io/projected/f28f0059-e25f-43ee-bab5-8cb570b14aae-kube-api-access-96x6r\") pod \"kube-storage-version-migrator-operator-7658d9b749-6zrvv\" (UID: \"f28f0059-e25f-43ee-bab5-8cb570b14aae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.241187 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.241155 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 08:23:07.241222 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.241196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f28f0059-e25f-43ee-bab5-8cb570b14aae-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6zrvv\" (UID: \"f28f0059-e25f-43ee-bab5-8cb570b14aae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.241257 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.241234 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:07.741214612 +0000 UTC m=+259.820252920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : secret "router-metrics-certs-default" not found Jul 21 08:23:07.241314 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.241275 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-stats-auth\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.241405 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.241364 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppt7\" (UniqueName: \"kubernetes.io/projected/bae2b6a9-8844-42aa-9c78-1d931caed9d4-kube-api-access-4ppt7\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.241555 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.241465 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.241617 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.241593 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:07.741578949 +0000 UTC m=+259.820617272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : configmap references non-existent config key: service-ca.crt Jul 21 08:23:07.241675 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.241628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f28f0059-e25f-43ee-bab5-8cb570b14aae-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6zrvv\" (UID: \"f28f0059-e25f-43ee-bab5-8cb570b14aae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.243515 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.243494 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f28f0059-e25f-43ee-bab5-8cb570b14aae-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6zrvv\" (UID: \"f28f0059-e25f-43ee-bab5-8cb570b14aae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.243629 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.243600 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-default-certificate\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.243778 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.243756 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-stats-auth\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.251553 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.251534 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppt7\" (UniqueName: \"kubernetes.io/projected/bae2b6a9-8844-42aa-9c78-1d931caed9d4-kube-api-access-4ppt7\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.251640 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.251566 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96x6r\" (UniqueName: \"kubernetes.io/projected/f28f0059-e25f-43ee-bab5-8cb570b14aae-kube-api-access-96x6r\") pod \"kube-storage-version-migrator-operator-7658d9b749-6zrvv\" (UID: \"f28f0059-e25f-43ee-bab5-8cb570b14aae\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.254542 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.254529 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-nxz4z" Jul 21 08:23:07.259539 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.259525 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" Jul 21 08:23:07.325181 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.325117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:07.348715 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.348685 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" Jul 21 08:23:07.388863 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.388824 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-nxz4z"] Jul 21 08:23:07.402260 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.402234 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw"] Jul 21 08:23:07.407002 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:07.406973 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72316d11_75cc_4fa6_8a04_f21232313286.slice/crio-bba50172415474265b68b31b367da9a9146407ca219ad0bdd4007b12f98f84f0 WatchSource:0}: Error finding container bba50172415474265b68b31b367da9a9146407ca219ad0bdd4007b12f98f84f0: Status 404 returned error can't find the container with id bba50172415474265b68b31b367da9a9146407ca219ad0bdd4007b12f98f84f0 Jul 21 08:23:07.483474 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.483399 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv"] Jul 21 08:23:07.486311 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:07.486290 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28f0059_e25f_43ee_bab5_8cb570b14aae.slice/crio-7cf4ae577c2b2b678d0ada0fc26589646c6e8e5517cfc45c893d33cc66377365 WatchSource:0}: Error finding container 7cf4ae577c2b2b678d0ada0fc26589646c6e8e5517cfc45c893d33cc66377365: Status 404 returned error can't find the container with id 7cf4ae577c2b2b678d0ada0fc26589646c6e8e5517cfc45c893d33cc66377365 Jul 21 08:23:07.645677 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.645647 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:07.645848 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.645764 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 08:23:07.645848 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.645818 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls podName:7e3135af-6d9a-4fdc-86c5-604b3e7e25e2 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:08.645804116 +0000 UTC m=+260.724842425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-d7t69" (UID: "7e3135af-6d9a-4fdc-86c5-604b3e7e25e2") : secret "samples-operator-tls" not found Jul 21 08:23:07.747088 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.747004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.747088 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:07.747051 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:07.747260 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.747150 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 08:23:07.747260 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.747207 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:08.747191476 +0000 UTC m=+260.826229785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : secret "router-metrics-certs-default" not found Jul 21 08:23:07.747260 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:07.747238 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:08.747224499 +0000 UTC m=+260.826262808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : configmap references non-existent config key: service-ca.crt Jul 21 08:23:08.141573 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:08.141524 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" event={"ID":"72316d11-75cc-4fa6-8a04-f21232313286","Type":"ContainerStarted","Data":"bba50172415474265b68b31b367da9a9146407ca219ad0bdd4007b12f98f84f0"} Jul 21 08:23:08.143195 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:08.143164 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" event={"ID":"f28f0059-e25f-43ee-bab5-8cb570b14aae","Type":"ContainerStarted","Data":"7cf4ae577c2b2b678d0ada0fc26589646c6e8e5517cfc45c893d33cc66377365"} Jul 21 08:23:08.144547 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:08.144525 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-nxz4z" event={"ID":"115156cb-6b04-420e-a460-b61a69906ef2","Type":"ContainerStarted","Data":"e9dda7d0b59acd17a62317fcd9b01d5a36926603acfb305794ba0bda850b34b6"} Jul 21 08:23:08.654055 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:08.654011 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:08.654241 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:08.654097 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 08:23:08.654241 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:08.654169 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls podName:7e3135af-6d9a-4fdc-86c5-604b3e7e25e2 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:10.654151575 +0000 UTC m=+262.733189889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-d7t69" (UID: "7e3135af-6d9a-4fdc-86c5-604b3e7e25e2") : secret "samples-operator-tls" not found Jul 21 08:23:08.755553 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:08.755514 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:08.755743 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:08.755582 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:08.755743 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:08.755739 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 08:23:08.755866 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:08.755792 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:10.755778181 +0000 UTC m=+262.834816490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : secret "router-metrics-certs-default" not found Jul 21 08:23:08.756199 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:08.756182 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:10.756165048 +0000 UTC m=+262.835203377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : configmap references non-existent config key: service-ca.crt Jul 21 08:23:09.731577 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:09.731537 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:09.832676 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:09.832645 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:10.671175 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:10.671139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:10.671353 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:10.671315 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 08:23:10.671429 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:10.671411 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls podName:7e3135af-6d9a-4fdc-86c5-604b3e7e25e2 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:14.671372062 +0000 UTC m=+266.750410374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-d7t69" (UID: "7e3135af-6d9a-4fdc-86c5-604b3e7e25e2") : secret "samples-operator-tls" not found Jul 21 08:23:10.772311 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:10.772281 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:10.772732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:10.772325 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:10.772732 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:10.772478 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 08:23:10.772732 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:10.772514 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:14.772492082 +0000 UTC m=+266.851530391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : configmap references non-existent config key: service-ca.crt Jul 21 08:23:10.772732 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:10.772547 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:14.772537064 +0000 UTC m=+266.851575372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : secret "router-metrics-certs-default" not found Jul 21 08:23:11.160797 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.160741 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" event={"ID":"f28f0059-e25f-43ee-bab5-8cb570b14aae","Type":"ContainerStarted","Data":"0873c0a12476b1b650f25941eb8c152d73769194eadf0b65d08aa1656c291f12"} Jul 21 08:23:11.162370 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.162342 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-nxz4z" event={"ID":"115156cb-6b04-420e-a460-b61a69906ef2","Type":"ContainerStarted","Data":"8bb740fe2fb300a31316cc58af8189c536a1672d777e251f5e53cf02d2951cfc"} Jul 21 08:23:11.163825 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.163806 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" event={"ID":"72316d11-75cc-4fa6-8a04-f21232313286","Type":"ContainerStarted","Data":"fb7f58e7179a62c7e693f4cf86df424956cb0924a506a0a092041932302ffad1"} Jul 21 08:23:11.177111 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.177065 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" podStartSLOduration=1.330677551 podStartE2EDuration="4.177050058s" totalImagesPullingTime="2.846372507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:11.176105215 +0000 UTC m=+263.255143547" watchObservedRunningTime="2026-07-21 08:23:11.177050058 +0000 UTC m=+263.256088390" Jul 21 08:23:11.192292 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.192240 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-nxz4z" podStartSLOduration=2.255254882 podStartE2EDuration="5.192221152s" totalImagesPullingTime="2.93696627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:11.19088914 +0000 UTC m=+263.269927495" watchObservedRunningTime="2026-07-21 08:23:11.192221152 +0000 UTC m=+263.271259484" Jul 21 08:23:11.204913 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.204852 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" podStartSLOduration=2.276043413 podStartE2EDuration="5.204832457s" totalImagesPullingTime="2.928789044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:11.204330183 +0000 UTC m=+263.283368515" watchObservedRunningTime="2026-07-21 08:23:11.204832457 +0000 UTC m=+263.283870789" Jul 21 08:23:11.592487 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.592413 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd"] Jul 21 08:23:11.596759 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.596724 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:11.596993 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.596884 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd" Jul 21 08:23:11.600291 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.600246 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 21 08:23:11.600291 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.600265 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pq9gz\"" Jul 21 08:23:11.600499 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.600246 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 21 08:23:11.604371 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.604348 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pgk7x"] Jul 21 08:23:11.608805 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.608098 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd"] Jul 21 08:23:11.608805 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.608209 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.614616 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.614404 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbchk\"" Jul 21 08:23:11.614616 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.614450 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 08:23:11.615610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.615591 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 08:23:11.641977 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.641951 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgk7x"] Jul 21 08:23:11.679890 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.679861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/65710829-45b3-467d-ba2c-5b3e02fde342-data-volume\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.680020 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.679906 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.680020 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.679973 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/65710829-45b3-467d-ba2c-5b3e02fde342-crio-socket\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.680309 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.680024 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.680309 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.680064 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hk6j\" (UniqueName: \"kubernetes.io/projected/49faf5d9-b52e-4787-b02f-81e90da5fb72-kube-api-access-5hk6j\") pod \"migrator-7cc9c7ffd7-24bvd\" (UID: \"49faf5d9-b52e-4787-b02f-81e90da5fb72\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd" Jul 21 08:23:11.680309 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.680152 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsnw\" (UniqueName: \"kubernetes.io/projected/65710829-45b3-467d-ba2c-5b3e02fde342-kube-api-access-wwsnw\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.704115 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.704088 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:11.761691 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.761659 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:11.780917 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.780893 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/65710829-45b3-467d-ba2c-5b3e02fde342-crio-socket\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.781234 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.780945 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.781234 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:11.781034 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:11.781234 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.781085 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hk6j\" (UniqueName: \"kubernetes.io/projected/49faf5d9-b52e-4787-b02f-81e90da5fb72-kube-api-access-5hk6j\") pod \"migrator-7cc9c7ffd7-24bvd\" (UID: \"49faf5d9-b52e-4787-b02f-81e90da5fb72\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd" Jul 21 08:23:11.781234 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.781027 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/65710829-45b3-467d-ba2c-5b3e02fde342-crio-socket\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.781234 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:11.781165 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:12.281146039 +0000 UTC m=+264.360184351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:11.781234 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.781225 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsnw\" (UniqueName: \"kubernetes.io/projected/65710829-45b3-467d-ba2c-5b3e02fde342-kube-api-access-wwsnw\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.781493 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.781248 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/65710829-45b3-467d-ba2c-5b3e02fde342-data-volume\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.781493 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.781269 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.781493 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:11.781343 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 08:23:11.781493 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:11.781402 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:12.281370421 +0000 UTC m=+264.360408730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : configmap "kube-rbac-proxy" not found Jul 21 08:23:11.781624 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.781591 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/65710829-45b3-467d-ba2c-5b3e02fde342-data-volume\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.789038 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.789007 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsnw\" (UniqueName: \"kubernetes.io/projected/65710829-45b3-467d-ba2c-5b3e02fde342-kube-api-access-wwsnw\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:11.789829 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.789813 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hk6j\" (UniqueName: \"kubernetes.io/projected/49faf5d9-b52e-4787-b02f-81e90da5fb72-kube-api-access-5hk6j\") pod \"migrator-7cc9c7ffd7-24bvd\" (UID: \"49faf5d9-b52e-4787-b02f-81e90da5fb72\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd" Jul 21 08:23:11.909294 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:11.909259 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd" Jul 21 08:23:12.041585 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.041553 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd"] Jul 21 08:23:12.044316 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:12.044293 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49faf5d9_b52e_4787_b02f_81e90da5fb72.slice/crio-f9c8e95af5368978ec73db7b646388d9637f0e453ef984a2fccaf1de18b4e58d WatchSource:0}: Error finding container f9c8e95af5368978ec73db7b646388d9637f0e453ef984a2fccaf1de18b4e58d: Status 404 returned error can't find the container with id f9c8e95af5368978ec73db7b646388d9637f0e453ef984a2fccaf1de18b4e58d Jul 21 08:23:12.166591 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.166489 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:12.167630 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.167609 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd" event={"ID":"49faf5d9-b52e-4787-b02f-81e90da5fb72","Type":"ContainerStarted","Data":"f9c8e95af5368978ec73db7b646388d9637f0e453ef984a2fccaf1de18b4e58d"} Jul 21 08:23:12.266676 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.266648 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:12.285858 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.285836 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:12.285989 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:12.285975 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:12.286132 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:12.286107 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:13.28606992 +0000 UTC m=+265.365108235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:12.286606 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.286263 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:12.286606 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:12.286317 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 08:23:12.286606 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:12.286360 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:13.286345796 +0000 UTC m=+265.365384104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : configmap "kube-rbac-proxy" not found Jul 21 08:23:12.390949 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.390919 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:12.446110 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.446023 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:12.870654 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.870568 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:12.971462 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:12.971421 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:13.294714 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:13.294679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:13.294922 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:13.294747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:13.294922 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:13.294829 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 08:23:13.294922 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:13.294863 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:13.294922 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:13.294916 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:15.294882878 +0000 UTC m=+267.373921196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : configmap "kube-rbac-proxy" not found Jul 21 08:23:13.295141 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:13.294936 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:15.294926295 +0000 UTC m=+267.373964613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:13.800274 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:13.800247 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-llppt_a59731f0-2456-4447-a672-69a180edc896/dns-node-resolver/0.log" Jul 21 08:23:14.175649 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.175619 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd" event={"ID":"49faf5d9-b52e-4787-b02f-81e90da5fb72","Type":"ContainerStarted","Data":"4f7be5ef4abbc86f22c3227fccb398a2a90ee3a990c4da563bd9363344e69f00"} Jul 21 08:23:14.176011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.175667 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd" event={"ID":"49faf5d9-b52e-4787-b02f-81e90da5fb72","Type":"ContainerStarted","Data":"1e3cf45abd99b6e8986b0dc2842f1b817bbc4b9cbd71714e00414967b6f1eb79"} Jul 21 08:23:14.189845 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.189799 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-24bvd" podStartSLOduration=1.829559476 podStartE2EDuration="3.189782471s" totalImagesPullingTime="1.360222995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:14.188508175 +0000 UTC m=+266.267546505" watchObservedRunningTime="2026-07-21 08:23:14.189782471 +0000 UTC m=+266.268820802" Jul 21 08:23:14.261424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.261377 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:14.494870 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.494792 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:14.560268 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.560234 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:14.599943 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.599918 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5m66_c044a2bf-898d-4e0c-b624-01d347f5d5f9/node-ca/0.log" Jul 21 08:23:14.706536 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.706499 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:14.706706 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:14.706611 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 08:23:14.706706 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:14.706675 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls podName:7e3135af-6d9a-4fdc-86c5-604b3e7e25e2 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:22.706659931 +0000 UTC m=+274.785698239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-d7t69" (UID: "7e3135af-6d9a-4fdc-86c5-604b3e7e25e2") : secret "samples-operator-tls" not found Jul 21 08:23:14.807796 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.807715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:14.807929 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:14.807822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:14.807929 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:14.807879 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 08:23:14.808016 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:14.807933 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:22.807916608 +0000 UTC m=+274.886954922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : configmap references non-existent config key: service-ca.crt Jul 21 08:23:14.808016 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:14.807955 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:22.807945926 +0000 UTC m=+274.886984241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : secret "router-metrics-certs-default" not found Jul 21 08:23:15.311711 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:15.311679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:15.312071 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:15.311814 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:15.312071 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:15.311848 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:15.312071 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:15.311900 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 08:23:15.312071 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:15.311913 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:19.311892979 +0000 UTC m=+271.390931309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:15.312071 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:15.311934 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:19.311924401 +0000 UTC m=+271.390962716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : configmap "kube-rbac-proxy" not found Jul 21 08:23:15.365599 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:15.365570 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:15.600411 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:15.600316 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8kwc_041c75cc-9740-48f5-8f2b-12043420de1e/serve-healthcheck-canary/0.log" Jul 21 08:23:17.866028 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:17.865999 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:18.131465 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:18.131356 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:18.218008 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:18.217976 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:18.317606 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:18.317578 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:18.400217 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:18.400183 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:18.547437 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:18.547402 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:19.346048 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:19.345997 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:19.346447 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:19.346161 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:19.346447 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:19.346167 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:19.346447 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:19.346224 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:27.346205507 +0000 UTC m=+279.425243817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:19.346447 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:19.346245 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 08:23:19.346447 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:19.346278 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:27.346267201 +0000 UTC m=+279.425305514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : configmap "kube-rbac-proxy" not found Jul 21 08:23:19.730999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:19.730962 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:19.923985 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:19.923954 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:20.121832 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:20.121753 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:20.323162 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:20.323132 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:20.578804 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:20.578776 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:21.537354 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:21.537312 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:21.641628 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:21.641588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:21.735860 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:21.735829 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:21.837643 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:21.837535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:21.937962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:21.937928 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:22.326856 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:22.326824 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:22.773628 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:22.773597 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:22.776069 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:22.776043 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3135af-6d9a-4fdc-86c5-604b3e7e25e2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-d7t69\" (UID: \"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:22.870281 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:22.870247 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" Jul 21 08:23:22.874667 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:22.874645 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:22.874713 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:22.874693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:22.874838 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:22.874823 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle podName:bae2b6a9-8844-42aa-9c78-1d931caed9d4 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:38.874801844 +0000 UTC m=+290.953840155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle") pod "router-default-8566ccfd7b-fdt46" (UID: "bae2b6a9-8844-42aa-9c78-1d931caed9d4") : configmap references non-existent config key: service-ca.crt Jul 21 08:23:22.877272 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:22.877247 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae2b6a9-8844-42aa-9c78-1d931caed9d4-metrics-certs\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:22.996634 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:22.996606 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69"] Jul 21 08:23:23.120068 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:23.120036 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:23.203009 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:23.202975 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" event={"ID":"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2","Type":"ContainerStarted","Data":"114ad3964c6076b351905d8dddf8ac0971cbe334454dc6e2312832b1590189e9"} Jul 21 08:23:23.321081 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:23.320994 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:23.800591 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:23.800562 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:23.921613 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:23.921564 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:24.330363 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:24.330334 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:24.487141 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:24.487100 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:24.762944 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:24.762901 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:25.130909 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:25.130800 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:25.210596 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:25.210569 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" event={"ID":"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2","Type":"ContainerStarted","Data":"436ddf32461d2986ac92b89146093cc90630252880d3a9af2860c29e7349284c"} Jul 21 08:23:25.210739 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:25.210616 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" event={"ID":"7e3135af-6d9a-4fdc-86c5-604b3e7e25e2","Type":"ContainerStarted","Data":"5d02bc59dd57051f52cd71e58a118b0340e4aa4c0e2349079fb52af18ef91bf0"} Jul 21 08:23:25.230999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:25.230951 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-d7t69" podStartSLOduration=17.490654511 podStartE2EDuration="19.230934935s" totalImagesPullingTime="1.740280424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:25.230089343 +0000 UTC m=+277.309127677" watchObservedRunningTime="2026-07-21 08:23:25.230934935 +0000 UTC m=+277.309973244" Jul 21 08:23:25.329803 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:25.329760 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:25.525779 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:25.525737 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:25.922401 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:25.922335 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:26.510239 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:26.510208 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:26.738120 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:26.738091 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:26.933529 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:26.933495 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:27.123408 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:27.123355 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:27.414466 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:27.414428 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:27.414649 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:27.414490 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:27.414649 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:27.414604 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:27.414754 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:27.414656 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls podName:65710829-45b3-467d-ba2c-5b3e02fde342 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:43.414642064 +0000 UTC m=+295.493680436 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgk7x" (UID: "65710829-45b3-467d-ba2c-5b3e02fde342") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:27.415075 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:27.415048 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65710829-45b3-467d-ba2c-5b3e02fde342-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:27.541101 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:27.541070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:28.126151 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:28.126119 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:28.342937 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:28.342900 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:28.531792 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:28.531758 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:29.905310 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:29.905264 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" podUID="f8bcd643-8b1e-4672-a68b-759c36bfad9d" Jul 21 08:23:29.905310 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:29.905288 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-75676588dc-89tmc" podUID="83c926ee-ff20-44a2-84da-31998e65e2fa" Jul 21 08:23:29.905310 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:29.905289 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xws6r" podUID="0f822e8b-5f1a-4335-8b98-928c1fb36ced" Jul 21 08:23:30.227966 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:30.227882 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:23:30.227966 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:30.227912 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xws6r" Jul 21 08:23:30.228166 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:30.228021 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:23:31.381213 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:31.381184 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:31.496881 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:31.496852 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:33.869886 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:33.869847 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:23:33.870369 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:33.869908 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:23:33.870369 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:33.869942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:23:33.872498 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:33.872472 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8bcd643-8b1e-4672-a68b-759c36bfad9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-kwwxv\" (UID: \"f8bcd643-8b1e-4672-a68b-759c36bfad9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:23:33.872599 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:33.872504 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f822e8b-5f1a-4335-8b98-928c1fb36ced-metrics-tls\") pod \"dns-default-xws6r\" (UID: \"0f822e8b-5f1a-4335-8b98-928c1fb36ced\") " pod="openshift-dns/dns-default-xws6r" Jul 21 08:23:33.872599 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:33.872540 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"image-registry-75676588dc-89tmc\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:23:34.131803 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:34.131713 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hvc2x\"" Jul 21 08:23:34.131803 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:34.131713 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rb9mm\"" Jul 21 08:23:34.131803 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:34.131772 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jw7kl\"" Jul 21 08:23:34.138979 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:34.138954 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:23:34.139082 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:34.138994 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xws6r" Jul 21 08:23:34.139082 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:34.138959 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" Jul 21 08:23:34.293830 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:34.293728 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75676588dc-89tmc"] Jul 21 08:23:34.298375 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:34.298351 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c926ee_ff20_44a2_84da_31998e65e2fa.slice/crio-ecc70614cbdbc504dfe1b0b10362631196175ad9cad6b61b63119ef714df0ba2 WatchSource:0}: Error finding container ecc70614cbdbc504dfe1b0b10362631196175ad9cad6b61b63119ef714df0ba2: Status 404 returned error can't find the container with id ecc70614cbdbc504dfe1b0b10362631196175ad9cad6b61b63119ef714df0ba2 Jul 21 08:23:34.305966 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:34.305944 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xws6r"] Jul 21 08:23:34.308221 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:34.308201 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f822e8b_5f1a_4335_8b98_928c1fb36ced.slice/crio-78cb879796cfa2a57185842086d6f7c3dac36066026a6b65bb8f62d530c04ece WatchSource:0}: Error finding container 78cb879796cfa2a57185842086d6f7c3dac36066026a6b65bb8f62d530c04ece: Status 404 returned error can't find the container with id 78cb879796cfa2a57185842086d6f7c3dac36066026a6b65bb8f62d530c04ece Jul 21 08:23:34.322640 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:34.322618 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv"] Jul 21 08:23:34.327461 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:34.327433 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bcd643_8b1e_4672_a68b_759c36bfad9d.slice/crio-88376e131ecd554565e67e474a6997c1a17fccb4b6201a093dc7edfc4f5c1a1e WatchSource:0}: Error finding container 88376e131ecd554565e67e474a6997c1a17fccb4b6201a093dc7edfc4f5c1a1e: Status 404 returned error can't find the container with id 88376e131ecd554565e67e474a6997c1a17fccb4b6201a093dc7edfc4f5c1a1e Jul 21 08:23:35.244807 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:35.244772 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" event={"ID":"f8bcd643-8b1e-4672-a68b-759c36bfad9d","Type":"ContainerStarted","Data":"88376e131ecd554565e67e474a6997c1a17fccb4b6201a093dc7edfc4f5c1a1e"} Jul 21 08:23:35.246777 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:35.246751 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75676588dc-89tmc" event={"ID":"83c926ee-ff20-44a2-84da-31998e65e2fa","Type":"ContainerStarted","Data":"c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5"} Jul 21 08:23:35.246902 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:35.246797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75676588dc-89tmc" event={"ID":"83c926ee-ff20-44a2-84da-31998e65e2fa","Type":"ContainerStarted","Data":"ecc70614cbdbc504dfe1b0b10362631196175ad9cad6b61b63119ef714df0ba2"} Jul 21 08:23:35.246902 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:35.246874 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:23:35.248203 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:35.248182 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xws6r" event={"ID":"0f822e8b-5f1a-4335-8b98-928c1fb36ced","Type":"ContainerStarted","Data":"78cb879796cfa2a57185842086d6f7c3dac36066026a6b65bb8f62d530c04ece"} Jul 21 08:23:35.287447 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:35.287345 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75676588dc-89tmc" podStartSLOduration=286.287323205 podStartE2EDuration="4m46.287323205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:35.285213563 +0000 UTC m=+287.364251896" watchObservedRunningTime="2026-07-21 08:23:35.287323205 +0000 UTC m=+287.366361536" Jul 21 08:23:36.256712 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:36.256666 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" event={"ID":"f8bcd643-8b1e-4672-a68b-759c36bfad9d","Type":"ContainerStarted","Data":"adc46c2c764c5c630bafed00152603714d697c98b9c99e625fa051f8b65cdfd7"} Jul 21 08:23:36.274280 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:36.274232 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-kwwxv" podStartSLOduration=263.968636405 podStartE2EDuration="4m25.274216076s" totalImagesPullingTime="1.305579671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:36.271998488 +0000 UTC m=+288.351036858" watchObservedRunningTime="2026-07-21 08:23:36.274216076 +0000 UTC m=+288.353254407" Jul 21 08:23:37.016861 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:37.016775 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:37.114652 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:37.114591 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:37.262564 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:37.262532 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xws6r" event={"ID":"0f822e8b-5f1a-4335-8b98-928c1fb36ced","Type":"ContainerStarted","Data":"89d5d189fea30f273ff6fb0f4c88fbd175f60ffc10d9823b58702b49940c53b8"} Jul 21 08:23:37.262965 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:37.262590 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xws6r" event={"ID":"0f822e8b-5f1a-4335-8b98-928c1fb36ced","Type":"ContainerStarted","Data":"2c275bb5ea382ead4c26b9b12463789cbeb699134bbb4ea6de9767e68cd568d0"} Jul 21 08:23:37.279961 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:37.279864 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xws6r" podStartSLOduration=251.946733514 podStartE2EDuration="4m14.279849895s" totalImagesPullingTime="2.333116381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:37.278326503 +0000 UTC m=+289.357364833" watchObservedRunningTime="2026-07-21 08:23:37.279849895 +0000 UTC m=+289.358888226" Jul 21 08:23:38.266439 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:38.266400 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xws6r" Jul 21 08:23:38.910213 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:38.910179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:38.910779 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:38.910759 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bae2b6a9-8844-42aa-9c78-1d931caed9d4-service-ca-bundle\") pod \"router-default-8566ccfd7b-fdt46\" (UID: \"bae2b6a9-8844-42aa-9c78-1d931caed9d4\") " pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:39.153778 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:39.153740 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:39.322106 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:39.322071 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8566ccfd7b-fdt46"] Jul 21 08:23:39.324769 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:39.324739 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae2b6a9_8844_42aa_9c78_1d931caed9d4.slice/crio-dafdcc0de4e0d232b3214944f7dbe8298751cd1aec631d178154b8a6ecd1b586 WatchSource:0}: Error finding container dafdcc0de4e0d232b3214944f7dbe8298751cd1aec631d178154b8a6ecd1b586: Status 404 returned error can't find the container with id dafdcc0de4e0d232b3214944f7dbe8298751cd1aec631d178154b8a6ecd1b586 Jul 21 08:23:40.042306 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:40.042277 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:40.275181 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:40.275140 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8566ccfd7b-fdt46" event={"ID":"bae2b6a9-8844-42aa-9c78-1d931caed9d4","Type":"ContainerStarted","Data":"3227d85eb4ebbc31e45ea4ec6ae33b96e85a82494bf289301f452cadb59e681b"} Jul 21 08:23:40.275181 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:40.275188 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8566ccfd7b-fdt46" event={"ID":"bae2b6a9-8844-42aa-9c78-1d931caed9d4","Type":"ContainerStarted","Data":"dafdcc0de4e0d232b3214944f7dbe8298751cd1aec631d178154b8a6ecd1b586"} Jul 21 08:23:40.295671 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:40.295577 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8566ccfd7b-fdt46" podStartSLOduration=33.295559423 podStartE2EDuration="33.295559423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:40.295025802 +0000 UTC m=+292.374064133" watchObservedRunningTime="2026-07-21 08:23:40.295559423 +0000 UTC m=+292.374597758" Jul 21 08:23:41.154718 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.154684 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:41.157086 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.157064 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:41.279144 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.279116 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:41.280303 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.280284 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8566ccfd7b-fdt46" Jul 21 08:23:41.530141 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.530055 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:41.628913 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.628882 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:41.698981 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.698943 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q"] Jul 21 08:23:41.701978 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.701962 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" Jul 21 08:23:41.705929 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.705912 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8fdwb\"" Jul 21 08:23:41.706220 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.706206 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 08:23:41.724310 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.724282 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q"] Jul 21 08:23:41.728872 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.728850 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:41.730873 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.730852 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c7dad7be-0b60-4a0e-9139-cef84cc4f391-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-k726q\" (UID: \"c7dad7be-0b60-4a0e-9139-cef84cc4f391\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" Jul 21 08:23:41.832272 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:41.832190 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c7dad7be-0b60-4a0e-9139-cef84cc4f391-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-k726q\" (UID: \"c7dad7be-0b60-4a0e-9139-cef84cc4f391\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" Jul 21 08:23:41.832432 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:41.832332 2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 21 08:23:41.832432 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:41.832410 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7dad7be-0b60-4a0e-9139-cef84cc4f391-tls-certificates podName:c7dad7be-0b60-4a0e-9139-cef84cc4f391 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:42.332378255 +0000 UTC m=+294.411416564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c7dad7be-0b60-4a0e-9139-cef84cc4f391-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-k726q" (UID: "c7dad7be-0b60-4a0e-9139-cef84cc4f391") : secret "prometheus-operator-admission-webhook-tls" not found Jul 21 08:23:42.209557 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:42.209523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:42.335875 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:42.335834 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c7dad7be-0b60-4a0e-9139-cef84cc4f391-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-k726q\" (UID: \"c7dad7be-0b60-4a0e-9139-cef84cc4f391\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" Jul 21 08:23:42.338376 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:42.338352 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c7dad7be-0b60-4a0e-9139-cef84cc4f391-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-k726q\" (UID: \"c7dad7be-0b60-4a0e-9139-cef84cc4f391\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" Jul 21 08:23:42.353411 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:42.353370 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7dgc9" podUID="19a14646-a2c6-4a63-8716-6fb7127293cf" Jul 21 08:23:42.610223 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:42.610129 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" Jul 21 08:23:42.733862 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:42.733831 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q"] Jul 21 08:23:42.737155 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:42.737128 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7dad7be_0b60_4a0e_9139_cef84cc4f391.slice/crio-42d53453f6be63c371a4d614175ef1a3ca7d34ef75ef9c3a3cabb4019e3c1d10 WatchSource:0}: Error finding container 42d53453f6be63c371a4d614175ef1a3ca7d34ef75ef9c3a3cabb4019e3c1d10: Status 404 returned error can't find the container with id 42d53453f6be63c371a4d614175ef1a3ca7d34ef75ef9c3a3cabb4019e3c1d10 Jul 21 08:23:43.289828 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:43.289790 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" event={"ID":"c7dad7be-0b60-4a0e-9139-cef84cc4f391","Type":"ContainerStarted","Data":"42d53453f6be63c371a4d614175ef1a3ca7d34ef75ef9c3a3cabb4019e3c1d10"} Jul 21 08:23:43.445588 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:43.445544 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:43.447898 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:43.447877 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65710829-45b3-467d-ba2c-5b3e02fde342-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgk7x\" (UID: \"65710829-45b3-467d-ba2c-5b3e02fde342\") " pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:43.719699 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:43.719667 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgk7x" Jul 21 08:23:44.006081 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.006010 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgk7x"] Jul 21 08:23:44.010579 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:44.010547 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65710829_45b3_467d_ba2c_5b3e02fde342.slice/crio-399611d232f25a46422f9b9e546f779b7c02b5351f890d6132950e2938814dc2 WatchSource:0}: Error finding container 399611d232f25a46422f9b9e546f779b7c02b5351f890d6132950e2938814dc2: Status 404 returned error can't find the container with id 399611d232f25a46422f9b9e546f779b7c02b5351f890d6132950e2938814dc2 Jul 21 08:23:44.294627 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.294544 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgk7x" event={"ID":"65710829-45b3-467d-ba2c-5b3e02fde342","Type":"ContainerStarted","Data":"399611d232f25a46422f9b9e546f779b7c02b5351f890d6132950e2938814dc2"} Jul 21 08:23:44.295938 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.295915 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" event={"ID":"c7dad7be-0b60-4a0e-9139-cef84cc4f391","Type":"ContainerStarted","Data":"c31b9bad849f2dbafacabda265698a5b4f07af6f79031cc005da8920a8cc42db"} Jul 21 08:23:44.296226 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.296206 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" Jul 21 08:23:44.301007 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.300988 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" Jul 21 08:23:44.310370 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.310327 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-k726q" podStartSLOduration=2.146254425 podStartE2EDuration="3.310313653s" totalImagesPullingTime="1.164059228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:44.30951971 +0000 UTC m=+296.388558042" watchObservedRunningTime="2026-07-21 08:23:44.310313653 +0000 UTC m=+296.389351986" Jul 21 08:23:44.715658 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.715533 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:44.758483 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.758453 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-td5t4"] Jul 21 08:23:44.760975 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.760858 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.764364 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.764342 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cfpm2\"" Jul 21 08:23:44.764508 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.764411 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 08:23:44.764569 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.764341 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 08:23:44.764692 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.764670 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 08:23:44.764749 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.764700 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 08:23:44.765140 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.764948 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 08:23:44.770143 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.770093 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-td5t4"] Jul 21 08:23:44.788211 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.788182 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:44.858752 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.858662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e86fc848-86bc-45a2-b378-1c543ff46a97-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.858752 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.858712 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e86fc848-86bc-45a2-b378-1c543ff46a97-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.858752 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.858755 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e86fc848-86bc-45a2-b378-1c543ff46a97-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.859064 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.858798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6p9\" (UniqueName: \"kubernetes.io/projected/e86fc848-86bc-45a2-b378-1c543ff46a97-kube-api-access-mj6p9\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.870923 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.870888 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:44.960047 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.960009 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e86fc848-86bc-45a2-b378-1c543ff46a97-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.960213 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.960061 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e86fc848-86bc-45a2-b378-1c543ff46a97-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.960213 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.960112 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e86fc848-86bc-45a2-b378-1c543ff46a97-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.960213 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.960157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6p9\" (UniqueName: \"kubernetes.io/projected/e86fc848-86bc-45a2-b378-1c543ff46a97-kube-api-access-mj6p9\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.960887 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.960859 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e86fc848-86bc-45a2-b378-1c543ff46a97-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.962841 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.962819 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e86fc848-86bc-45a2-b378-1c543ff46a97-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.962841 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.962830 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e86fc848-86bc-45a2-b378-1c543ff46a97-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:44.968965 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:44.968900 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6p9\" (UniqueName: \"kubernetes.io/projected/e86fc848-86bc-45a2-b378-1c543ff46a97-kube-api-access-mj6p9\") pod \"prometheus-operator-5945b5cfd-td5t4\" (UID: \"e86fc848-86bc-45a2-b378-1c543ff46a97\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:45.075671 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:45.075635 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" Jul 21 08:23:45.223091 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:45.223012 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-td5t4"] Jul 21 08:23:45.226776 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:45.226748 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86fc848_86bc_45a2_b378_1c543ff46a97.slice/crio-8097542e608ceb3d17fdce2de243b8ef3cfe89823c18aec4b12e784b4c14e3d2 WatchSource:0}: Error finding container 8097542e608ceb3d17fdce2de243b8ef3cfe89823c18aec4b12e784b4c14e3d2: Status 404 returned error can't find the container with id 8097542e608ceb3d17fdce2de243b8ef3cfe89823c18aec4b12e784b4c14e3d2 Jul 21 08:23:45.301529 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:45.301492 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" event={"ID":"e86fc848-86bc-45a2-b378-1c543ff46a97","Type":"ContainerStarted","Data":"8097542e608ceb3d17fdce2de243b8ef3cfe89823c18aec4b12e784b4c14e3d2"} Jul 21 08:23:48.272525 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:48.272494 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xws6r" Jul 21 08:23:49.228778 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:49.228759 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 08:23:49.318470 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:49.317811 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" event={"ID":"e86fc848-86bc-45a2-b378-1c543ff46a97","Type":"ContainerStarted","Data":"e872bed8516235aab833e4caeec762299df135c0e2ca662fe6f5af3aed42e2b2"} Jul 21 08:23:49.555898 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:49.555814 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:50.323545 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:50.323502 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgk7x" event={"ID":"65710829-45b3-467d-ba2c-5b3e02fde342","Type":"ContainerStarted","Data":"1d2006f31c6f7cc1e388dd599df7cdec991e71642db1cf38a6e9823780cdc3b4"} Jul 21 08:23:50.325145 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:50.325119 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" event={"ID":"e86fc848-86bc-45a2-b378-1c543ff46a97","Type":"ContainerStarted","Data":"4cde196e0551602a43f6917569e99314e042c152ede08d82bffb44b2eed40a9b"} Jul 21 08:23:50.343787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:50.343747 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-td5t4" podStartSLOduration=2.360753427 podStartE2EDuration="6.34373258s" totalImagesPullingTime="3.982979153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:50.342529408 +0000 UTC m=+302.421567738" watchObservedRunningTime="2026-07-21 08:23:50.34373258 +0000 UTC m=+302.422770927" Jul 21 08:23:51.894766 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:51.894670 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:52.136717 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.136682 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zndk7"] Jul 21 08:23:52.160896 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.160828 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.163520 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.163496 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x4bjv\"" Jul 21 08:23:52.163686 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.163569 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 08:23:52.163791 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.163773 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 08:23:52.163888 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.163857 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 08:23:52.224804 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.224770 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-textfile\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.224992 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.224827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-tls\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.224992 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.224858 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7c9v\" (UniqueName: \"kubernetes.io/projected/fc259ca0-b711-40b9-9c06-4ec62ca949f9-kube-api-access-x7c9v\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.224992 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.224918 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-wtmp\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.224992 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.224943 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.225163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.225032 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc259ca0-b711-40b9-9c06-4ec62ca949f9-root\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.225163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.225084 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc259ca0-b711-40b9-9c06-4ec62ca949f9-sys\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.225163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.225117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.225163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.225153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc259ca0-b711-40b9-9c06-4ec62ca949f9-metrics-client-ca\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.325995 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.325968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-tls\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326151 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.325999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7c9v\" (UniqueName: \"kubernetes.io/projected/fc259ca0-b711-40b9-9c06-4ec62ca949f9-kube-api-access-x7c9v\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326151 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326031 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-wtmp\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326151 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326151 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:52.326128 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 08:23:52.326300 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326154 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc259ca0-b711-40b9-9c06-4ec62ca949f9-root\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326300 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326163 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-wtmp\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326300 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:23:52.326193 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-tls podName:fc259ca0-b711-40b9-9c06-4ec62ca949f9 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:52.826174394 +0000 UTC m=+304.905212719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-tls") pod "node-exporter-zndk7" (UID: "fc259ca0-b711-40b9-9c06-4ec62ca949f9") : secret "node-exporter-tls" not found Jul 21 08:23:52.326300 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326205 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc259ca0-b711-40b9-9c06-4ec62ca949f9-root\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326300 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326241 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc259ca0-b711-40b9-9c06-4ec62ca949f9-sys\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326300 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326623 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc259ca0-b711-40b9-9c06-4ec62ca949f9-metrics-client-ca\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326623 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326329 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc259ca0-b711-40b9-9c06-4ec62ca949f9-sys\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326623 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-textfile\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326817 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326665 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-textfile\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326817 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326713 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.326911 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.326842 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc259ca0-b711-40b9-9c06-4ec62ca949f9-metrics-client-ca\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.328677 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.328656 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.335492 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.335467 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7c9v\" (UniqueName: \"kubernetes.io/projected/fc259ca0-b711-40b9-9c06-4ec62ca949f9-kube-api-access-x7c9v\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.830723 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.830685 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-tls\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:52.833474 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:52.833453 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc259ca0-b711-40b9-9c06-4ec62ca949f9-node-exporter-tls\") pod \"node-exporter-zndk7\" (UID: \"fc259ca0-b711-40b9-9c06-4ec62ca949f9\") " pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:53.073364 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.073339 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zndk7" Jul 21 08:23:53.084436 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:53.084411 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc259ca0_b711_40b9_9c06_4ec62ca949f9.slice/crio-ad1eb4b1d131e72899d1de5594b2a3a6caf4d254389409d2a8680ec00e956281 WatchSource:0}: Error finding container ad1eb4b1d131e72899d1de5594b2a3a6caf4d254389409d2a8680ec00e956281: Status 404 returned error can't find the container with id ad1eb4b1d131e72899d1de5594b2a3a6caf4d254389409d2a8680ec00e956281 Jul 21 08:23:53.086130 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.086113 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:23:53.198535 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.198502 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:53.292362 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.292330 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:53.337733 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.337648 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgk7x" event={"ID":"65710829-45b3-467d-ba2c-5b3e02fde342","Type":"ContainerStarted","Data":"78f1fd970c9d2602eda76402c6e075d4989602ae364c61b4f3e1d86c92c3de98"} Jul 21 08:23:53.338952 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.338914 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zndk7" event={"ID":"fc259ca0-b711-40b9-9c06-4ec62ca949f9","Type":"ContainerStarted","Data":"ad1eb4b1d131e72899d1de5594b2a3a6caf4d254389409d2a8680ec00e956281"} Jul 21 08:23:53.393714 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.393677 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:53.494186 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.494145 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:53.605805 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.605658 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:53.717727 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:53.717669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:54.144986 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:54.144954 2563 patch_prober.go:28] interesting pod/image-registry-75676588dc-89tmc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 21 08:23:54.145364 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:54.145030 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-75676588dc-89tmc" podUID="83c926ee-ff20-44a2-84da-31998e65e2fa" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 08:23:54.346842 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:54.346756 2563 generic.go:356] "Generic (PLEG): container finished" podID="fc259ca0-b711-40b9-9c06-4ec62ca949f9" containerID="ce4362d2ec6b47726ad2063f2e480c901d58e7232360e38fbf52c9b422cec518" exitCode=0 Jul 21 08:23:54.347012 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:54.346849 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zndk7" event={"ID":"fc259ca0-b711-40b9-9c06-4ec62ca949f9","Type":"ContainerDied","Data":"ce4362d2ec6b47726ad2063f2e480c901d58e7232360e38fbf52c9b422cec518"} Jul 21 08:23:55.149186 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:55.149154 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:55.256603 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:55.256562 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:55.354303 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:55.354272 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zndk7" event={"ID":"fc259ca0-b711-40b9-9c06-4ec62ca949f9","Type":"ContainerStarted","Data":"773c9bd50ed41fbbd7752b5b9d1b943fd1154726724ee01eda14ac92c23e0da7"} Jul 21 08:23:55.354512 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:55.354337 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zndk7" event={"ID":"fc259ca0-b711-40b9-9c06-4ec62ca949f9","Type":"ContainerStarted","Data":"9304d83016e3fa4aa0f00c210a4bbbdb29bab217200f592827df451b8c002b0b"} Jul 21 08:23:55.705807 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:55.705770 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:56.262248 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.262222 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:23:56.295578 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.295525 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zndk7" podStartSLOduration=3.263526521 podStartE2EDuration="4.295502341s" totalImagesPullingTime="950.932901ms" totalInitContainerRuntime="81.042919ms" isStatefulPod=true podCreationTimestamp="2026-07-21 08:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:55.381687813 +0000 UTC m=+307.460726145" watchObservedRunningTime="2026-07-21 08:23:56.295502341 +0000 UTC m=+308.374540671" Jul 21 08:23:56.358920 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.358894 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:23:56.456577 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.456543 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5dd65855c4-dfhcn"] Jul 21 08:23:56.483943 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.483916 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dd65855c4-dfhcn"] Jul 21 08:23:56.484109 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.484044 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.488676 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.488648 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 08:23:56.488811 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.488752 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-foo2f8i5smvie\"" Jul 21 08:23:56.489486 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.489443 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 08:23:56.489967 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.489699 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 08:23:56.489967 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.489725 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ppxhj\"" Jul 21 08:23:56.489967 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.489823 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 08:23:56.517003 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.516939 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:56.564908 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.564880 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-audit-log\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.565052 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.564923 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-metrics-server-audit-profiles\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.565052 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.564969 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxvjb\" (UniqueName: \"kubernetes.io/projected/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-kube-api-access-kxvjb\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.565052 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.564993 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.565052 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.565011 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-secret-metrics-server-tls\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.565052 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.565035 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-client-ca-bundle\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.565207 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.565092 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-secret-metrics-server-client-certs\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.623549 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.623527 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:56.665930 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.665905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-metrics-server-audit-profiles\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.666052 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.665971 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxvjb\" (UniqueName: \"kubernetes.io/projected/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-kube-api-access-kxvjb\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.666052 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.666008 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.666052 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.666037 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-secret-metrics-server-tls\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.666210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.666065 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-client-ca-bundle\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.666210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.666094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-secret-metrics-server-client-certs\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.666210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.666153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-audit-log\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.666614 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.666588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-audit-log\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.666807 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.666746 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.667317 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.667291 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-metrics-server-audit-profiles\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.668741 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.668721 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-client-ca-bundle\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.668932 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.668909 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-secret-metrics-server-tls\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.668973 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.668959 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-secret-metrics-server-client-certs\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.675354 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.675333 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxvjb\" (UniqueName: \"kubernetes.io/projected/1fb3f2f4-48d4-4918-9aef-f4312bcd9f89-kube-api-access-kxvjb\") pod \"metrics-server-5dd65855c4-dfhcn\" (UID: \"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89\") " pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.795189 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.795097 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:23:56.914205 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.914176 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:56.943649 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:56.943603 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dd65855c4-dfhcn"] Jul 21 08:23:56.946375 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:23:56.946349 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb3f2f4_48d4_4918_9aef_f4312bcd9f89.slice/crio-79aee10ab1a398b05c8e6c066e6d91500838ccb5f72f074b46071e574ee2b1f3 WatchSource:0}: Error finding container 79aee10ab1a398b05c8e6c066e6d91500838ccb5f72f074b46071e574ee2b1f3: Status 404 returned error can't find the container with id 79aee10ab1a398b05c8e6c066e6d91500838ccb5f72f074b46071e574ee2b1f3 Jul 21 08:23:57.283642 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:57.283615 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:57.363018 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:57.362976 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" event={"ID":"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89","Type":"ContainerStarted","Data":"79aee10ab1a398b05c8e6c066e6d91500838ccb5f72f074b46071e574ee2b1f3"} Jul 21 08:23:57.364960 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:57.364936 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgk7x" event={"ID":"65710829-45b3-467d-ba2c-5b3e02fde342","Type":"ContainerStarted","Data":"9b749dc2c27568652f6fd199d061145a4a4afc5198422bae56a3ba2b89d648b3"} Jul 21 08:23:57.398067 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:57.394546 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pgk7x" podStartSLOduration=33.923954214 podStartE2EDuration="46.394525983s" totalImagesPullingTime="12.470571769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:23:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:57.392746372 +0000 UTC m=+309.471784703" watchObservedRunningTime="2026-07-21 08:23:57.394525983 +0000 UTC m=+309.473564315" Jul 21 08:23:57.516756 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:57.516723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:57.614884 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:57.614798 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:58.445009 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:58.444949 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:58.805072 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:58.804975 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75676588dc-89tmc"] Jul 21 08:23:58.941551 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:58.941519 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:59.042128 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:59.042099 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:59.140173 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:59.140134 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:23:59.373332 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:59.373300 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" event={"ID":"1fb3f2f4-48d4-4918-9aef-f4312bcd9f89","Type":"ContainerStarted","Data":"97617552863f0665bd3099ac4ff8d5ef462a8dd6faa69f0d8f5a063d426cda57"} Jul 21 08:23:59.392198 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:23:59.392096 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" podStartSLOduration=1.7427925869999998 podStartE2EDuration="3.39207558s" totalImagesPullingTime="1.649282993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:59.391032852 +0000 UTC m=+311.470071169" watchObservedRunningTime="2026-07-21 08:23:59.39207558 +0000 UTC m=+311.471113914" Jul 21 08:24:05.104118 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:05.104088 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:05.172703 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:05.172674 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:05.273037 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:05.273008 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:05.393122 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:05.393081 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:05.794039 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:05.793948 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:05.844994 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:05.844967 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:05.965148 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:05.965117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:06.176623 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:06.176590 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:06.372428 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:06.372356 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:06.469080 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:06.468980 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:06.783817 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:06.783733 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:06.884476 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:06.884412 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:06.985351 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:06.985318 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:07.793734 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:07.793684 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:07.848537 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:07.848507 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:08.150013 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:08.149953 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:08.254320 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:08.254277 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:11.112316 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:11.112281 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:11.213465 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:11.213432 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:13.425044 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.425012 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-744565fbd-965pc"] Jul 21 08:24:13.429592 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.429565 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.432245 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.432224 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 08:24:13.432363 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.432268 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 08:24:13.432561 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.432543 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 08:24:13.433189 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.433174 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 08:24:13.433280 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.433244 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 08:24:13.433521 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.433502 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 08:24:13.433618 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.433511 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 08:24:13.433618 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.433579 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-js8zn\"" Jul 21 08:24:13.439503 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.439483 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744565fbd-965pc"] Jul 21 08:24:13.519799 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.519765 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-serving-cert\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.519948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.519802 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-oauth-config\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.519948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.519833 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-config\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.519948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.519912 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-service-ca\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.520046 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.519998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzl4p\" (UniqueName: \"kubernetes.io/projected/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-kube-api-access-hzl4p\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.520046 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.520022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-oauth-serving-cert\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.620692 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.620657 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzl4p\" (UniqueName: \"kubernetes.io/projected/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-kube-api-access-hzl4p\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.620866 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.620704 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-oauth-serving-cert\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.620866 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.620742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-serving-cert\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.620866 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.620785 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-oauth-config\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.620866 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.620813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-config\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.620866 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.620861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-service-ca\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.621516 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.621491 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-config\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.621621 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.621497 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-oauth-serving-cert\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.621799 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.621778 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-service-ca\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.623290 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.623261 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-oauth-config\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.623479 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.623459 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-serving-cert\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.630021 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.630000 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzl4p\" (UniqueName: \"kubernetes.io/projected/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-kube-api-access-hzl4p\") pod \"console-744565fbd-965pc\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.739306 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.739224 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:13.869791 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:13.869763 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744565fbd-965pc"] Jul 21 08:24:13.872889 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:24:13.872863 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24a4c8f_749e_4b6b_bb5d_cc8c0648fc75.slice/crio-beccab31bc4f510b2510a1928144626020e079241690cdf878fbb4b85926ac00 WatchSource:0}: Error finding container beccab31bc4f510b2510a1928144626020e079241690cdf878fbb4b85926ac00: Status 404 returned error can't find the container with id beccab31bc4f510b2510a1928144626020e079241690cdf878fbb4b85926ac00 Jul 21 08:24:14.428095 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:14.428063 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744565fbd-965pc" event={"ID":"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75","Type":"ContainerStarted","Data":"beccab31bc4f510b2510a1928144626020e079241690cdf878fbb4b85926ac00"} Jul 21 08:24:15.409889 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:15.409850 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:15.514021 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:15.513986 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:16.395831 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:16.395787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:16.438094 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:16.438059 2563 generic.go:356] "Generic (PLEG): container finished" podID="f28f0059-e25f-43ee-bab5-8cb570b14aae" containerID="0873c0a12476b1b650f25941eb8c152d73769194eadf0b65d08aa1656c291f12" exitCode=0 Jul 21 08:24:16.438257 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:16.438141 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" event={"ID":"f28f0059-e25f-43ee-bab5-8cb570b14aae","Type":"ContainerDied","Data":"0873c0a12476b1b650f25941eb8c152d73769194eadf0b65d08aa1656c291f12"} Jul 21 08:24:16.438593 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:16.438572 2563 scope.go:117] "RemoveContainer" containerID="0873c0a12476b1b650f25941eb8c152d73769194eadf0b65d08aa1656c291f12" Jul 21 08:24:16.497141 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:16.497099 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:16.795456 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:16.795326 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:24:16.795456 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:16.795398 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:24:17.213651 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:17.213623 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:17.314976 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:17.314949 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:17.450107 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:17.450067 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744565fbd-965pc" event={"ID":"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75","Type":"ContainerStarted","Data":"fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3"} Jul 21 08:24:17.452157 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:17.452136 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6zrvv" event={"ID":"f28f0059-e25f-43ee-bab5-8cb570b14aae","Type":"ContainerStarted","Data":"416ed534105d20a044f369b05c20fa9d6cffff47dc64a71ce952f624068d806e"} Jul 21 08:24:17.468091 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:17.467998 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-744565fbd-965pc" podStartSLOduration=1.053306193 podStartE2EDuration="4.467984134s" totalImagesPullingTime="3.414677941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:24:17.466815742 +0000 UTC m=+329.545854074" watchObservedRunningTime="2026-07-21 08:24:17.467984134 +0000 UTC m=+329.547022465" Jul 21 08:24:19.212253 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:19.212219 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:19.313110 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:19.313081 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:20.991174 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:20.991136 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fb9c5b84-pcl6q"] Jul 21 08:24:21.006297 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.006268 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb9c5b84-pcl6q"] Jul 21 08:24:21.006461 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.006370 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.014651 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.014631 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 08:24:21.186148 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.186114 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-serving-cert\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.186461 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.186436 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hc6\" (UniqueName: \"kubernetes.io/projected/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-kube-api-access-69hc6\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.186622 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.186606 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-service-ca\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.186754 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.186736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-trusted-ca-bundle\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.186832 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.186817 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-oauth-config\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.186885 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.186853 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-config\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.186885 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.186876 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-oauth-serving-cert\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.288294 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.288217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hc6\" (UniqueName: \"kubernetes.io/projected/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-kube-api-access-69hc6\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.288294 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.288277 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-service-ca\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.288481 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.288436 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-trusted-ca-bundle\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.288517 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.288492 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-oauth-config\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.288565 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.288518 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-config\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.288565 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.288540 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-oauth-serving-cert\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.288663 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.288570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-serving-cert\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.289067 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.289039 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-service-ca\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.289239 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.289215 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-oauth-serving-cert\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.289295 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.289240 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-trusted-ca-bundle\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.289412 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.289273 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-config\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.290982 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.290965 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-oauth-config\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.291183 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.291163 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-serving-cert\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.295893 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.295874 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hc6\" (UniqueName: \"kubernetes.io/projected/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-kube-api-access-69hc6\") pod \"console-6fb9c5b84-pcl6q\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.316771 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.316748 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:21.467115 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.466914 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb9c5b84-pcl6q"] Jul 21 08:24:21.469063 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.469032 2563 generic.go:356] "Generic (PLEG): container finished" podID="115156cb-6b04-420e-a460-b61a69906ef2" containerID="8bb740fe2fb300a31316cc58af8189c536a1672d777e251f5e53cf02d2951cfc" exitCode=0 Jul 21 08:24:21.469172 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.469100 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-nxz4z" event={"ID":"115156cb-6b04-420e-a460-b61a69906ef2","Type":"ContainerDied","Data":"8bb740fe2fb300a31316cc58af8189c536a1672d777e251f5e53cf02d2951cfc"} Jul 21 08:24:21.469467 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:21.469450 2563 scope.go:117] "RemoveContainer" containerID="8bb740fe2fb300a31316cc58af8189c536a1672d777e251f5e53cf02d2951cfc" Jul 21 08:24:21.469854 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:24:21.469829 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fbe7bf_71ae_404d_a5ec_b788ecdc02eb.slice/crio-cf0c2a5806d22e89c962c93c98ae1ff89fadef28eb83ba31920dd7b4209e860d WatchSource:0}: Error finding container cf0c2a5806d22e89c962c93c98ae1ff89fadef28eb83ba31920dd7b4209e860d: Status 404 returned error can't find the container with id cf0c2a5806d22e89c962c93c98ae1ff89fadef28eb83ba31920dd7b4209e860d Jul 21 08:24:22.210080 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:22.210045 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:22.273523 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:22.273498 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:22.474580 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:22.474495 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb9c5b84-pcl6q" event={"ID":"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb","Type":"ContainerStarted","Data":"ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698"} Jul 21 08:24:22.474580 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:22.474544 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb9c5b84-pcl6q" event={"ID":"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb","Type":"ContainerStarted","Data":"cf0c2a5806d22e89c962c93c98ae1ff89fadef28eb83ba31920dd7b4209e860d"} Jul 21 08:24:22.476213 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:22.476190 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-nxz4z" event={"ID":"115156cb-6b04-420e-a460-b61a69906ef2","Type":"ContainerStarted","Data":"5902b65c2032fbdeb0511a3132879cb02fef80427e2784b0168e11114d124755"} Jul 21 08:24:22.491423 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:22.491359 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fb9c5b84-pcl6q" podStartSLOduration=2.4913453309999998 podStartE2EDuration="2.491345331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:24:22.490215856 +0000 UTC m=+334.569254188" watchObservedRunningTime="2026-07-21 08:24:22.491345331 +0000 UTC m=+334.570383702" Jul 21 08:24:23.740114 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:23.740080 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:23.740582 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:23.740162 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:23.745840 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:23.745811 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:23.827198 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:23.827165 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75676588dc-89tmc" podUID="83c926ee-ff20-44a2-84da-31998e65e2fa" containerName="registry" containerID="cri-o://c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5" gracePeriod=30 Jul 21 08:24:24.076350 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.076328 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:24:24.107930 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.107903 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-certificates\") pod \"83c926ee-ff20-44a2-84da-31998e65e2fa\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " Jul 21 08:24:24.108067 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.107988 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-image-registry-private-configuration\") pod \"83c926ee-ff20-44a2-84da-31998e65e2fa\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " Jul 21 08:24:24.108067 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.108038 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-trusted-ca\") pod \"83c926ee-ff20-44a2-84da-31998e65e2fa\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " Jul 21 08:24:24.108067 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.108058 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-installation-pull-secrets\") pod \"83c926ee-ff20-44a2-84da-31998e65e2fa\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " Jul 21 08:24:24.108246 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.108103 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83c926ee-ff20-44a2-84da-31998e65e2fa-ca-trust-extracted\") pod \"83c926ee-ff20-44a2-84da-31998e65e2fa\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " Jul 21 08:24:24.108246 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.108149 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzbdz\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-kube-api-access-kzbdz\") pod \"83c926ee-ff20-44a2-84da-31998e65e2fa\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " Jul 21 08:24:24.108246 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.108182 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") pod \"83c926ee-ff20-44a2-84da-31998e65e2fa\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " Jul 21 08:24:24.108246 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.108218 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "83c926ee-ff20-44a2-84da-31998e65e2fa" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:24:24.108475 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.108459 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-certificates\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:24.108537 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.108476 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "83c926ee-ff20-44a2-84da-31998e65e2fa" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:24:24.110935 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.110906 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-kube-api-access-kzbdz" (OuterVolumeSpecName: "kube-api-access-kzbdz") pod "83c926ee-ff20-44a2-84da-31998e65e2fa" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa"). InnerVolumeSpecName "kube-api-access-kzbdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:24:24.111046 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.110908 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "83c926ee-ff20-44a2-84da-31998e65e2fa" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:24:24.111128 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.111046 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "83c926ee-ff20-44a2-84da-31998e65e2fa" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:24:24.111128 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.111108 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "83c926ee-ff20-44a2-84da-31998e65e2fa" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:24:24.116977 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.116957 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c926ee-ff20-44a2-84da-31998e65e2fa-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "83c926ee-ff20-44a2-84da-31998e65e2fa" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:24:24.209576 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.209545 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-bound-sa-token\") pod \"83c926ee-ff20-44a2-84da-31998e65e2fa\" (UID: \"83c926ee-ff20-44a2-84da-31998e65e2fa\") " Jul 21 08:24:24.209747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.209734 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-image-registry-private-configuration\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:24.209787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.209750 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83c926ee-ff20-44a2-84da-31998e65e2fa-trusted-ca\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:24.209787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.209759 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83c926ee-ff20-44a2-84da-31998e65e2fa-installation-pull-secrets\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:24.209787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.209768 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83c926ee-ff20-44a2-84da-31998e65e2fa-ca-trust-extracted\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:24.209787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.209776 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzbdz\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-kube-api-access-kzbdz\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:24.209787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.209784 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-registry-tls\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:24.211721 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.211700 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "83c926ee-ff20-44a2-84da-31998e65e2fa" (UID: "83c926ee-ff20-44a2-84da-31998e65e2fa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:24:24.310759 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.310685 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83c926ee-ff20-44a2-84da-31998e65e2fa-bound-sa-token\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:24.486453 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.486425 2563 generic.go:356] "Generic (PLEG): container finished" podID="83c926ee-ff20-44a2-84da-31998e65e2fa" containerID="c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5" exitCode=0 Jul 21 08:24:24.486617 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.486478 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75676588dc-89tmc" Jul 21 08:24:24.486617 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.486517 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75676588dc-89tmc" event={"ID":"83c926ee-ff20-44a2-84da-31998e65e2fa","Type":"ContainerDied","Data":"c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5"} Jul 21 08:24:24.486617 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.486572 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75676588dc-89tmc" event={"ID":"83c926ee-ff20-44a2-84da-31998e65e2fa","Type":"ContainerDied","Data":"ecc70614cbdbc504dfe1b0b10362631196175ad9cad6b61b63119ef714df0ba2"} Jul 21 08:24:24.486617 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.486597 2563 scope.go:117] "RemoveContainer" containerID="c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5" Jul 21 08:24:24.493457 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.493362 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:24.499655 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.499640 2563 scope.go:117] "RemoveContainer" containerID="c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5" Jul 21 08:24:24.499929 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:24:24.499907 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5\": container with ID starting with c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5 not found: ID does not exist" containerID="c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5" Jul 21 08:24:24.500002 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.499940 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5"} err="failed to get container status \"c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5\": rpc error: code = NotFound desc = could not find container \"c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5\": container with ID starting with c4e51075cd2f8e32aa01c7c4d40d21b83c070d481f71b21ede56edc3261627a5 not found: ID does not exist" Jul 21 08:24:24.528646 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.528622 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75676588dc-89tmc"] Jul 21 08:24:24.534181 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:24.534160 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75676588dc-89tmc"] Jul 21 08:24:25.217747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:25.217715 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:25.274836 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:25.274787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:26.205691 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:26.205654 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:26.257915 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:26.257871 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:26.357616 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:26.357586 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c926ee-ff20-44a2-84da-31998e65e2fa" path="/var/lib/kubelet/pods/83c926ee-ff20-44a2-84da-31998e65e2fa/volumes" Jul 21 08:24:31.317663 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.317629 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:31.318012 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.317691 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:31.322486 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.322458 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:31.354188 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.354168 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:31.410582 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.410556 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:31.514373 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.514292 2563 generic.go:356] "Generic (PLEG): container finished" podID="72316d11-75cc-4fa6-8a04-f21232313286" containerID="fb7f58e7179a62c7e693f4cf86df424956cb0924a506a0a092041932302ffad1" exitCode=0 Jul 21 08:24:31.514373 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.514365 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" event={"ID":"72316d11-75cc-4fa6-8a04-f21232313286","Type":"ContainerDied","Data":"fb7f58e7179a62c7e693f4cf86df424956cb0924a506a0a092041932302ffad1"} Jul 21 08:24:31.514838 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.514820 2563 scope.go:117] "RemoveContainer" containerID="fb7f58e7179a62c7e693f4cf86df424956cb0924a506a0a092041932302ffad1" Jul 21 08:24:31.518662 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.518643 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:24:31.581592 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.581567 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744565fbd-965pc"] Jul 21 08:24:31.753405 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:31.753344 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:32.519406 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:32.519346 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-czhgw" event={"ID":"72316d11-75cc-4fa6-8a04-f21232313286","Type":"ContainerStarted","Data":"dba60a64d7d287eda2f32b203f5eabc7a5adadaf965a8716777fc6e545559db6"} Jul 21 08:24:36.800333 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:36.800301 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:24:36.804291 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:36.804268 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5dd65855c4-dfhcn" Jul 21 08:24:53.697279 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:53.697243 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:24:56.609521 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.609443 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-744565fbd-965pc" podUID="c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" containerName="console" containerID="cri-o://fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3" gracePeriod=15 Jul 21 08:24:56.850408 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.850371 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:56.975482 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.975445 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-service-ca\") pod \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " Jul 21 08:24:56.975653 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.975495 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-serving-cert\") pod \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " Jul 21 08:24:56.975653 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.975532 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-oauth-config\") pod \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " Jul 21 08:24:56.975653 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.975547 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-config\") pod \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " Jul 21 08:24:56.975653 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.975568 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzl4p\" (UniqueName: \"kubernetes.io/projected/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-kube-api-access-hzl4p\") pod \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " Jul 21 08:24:56.975653 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.975596 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-oauth-serving-cert\") pod \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\" (UID: \"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75\") " Jul 21 08:24:56.976020 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.975990 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-service-ca" (OuterVolumeSpecName: "service-ca") pod "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" (UID: "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:24:56.976110 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.975996 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-config" (OuterVolumeSpecName: "console-config") pod "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" (UID: "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:24:56.976110 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.976051 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" (UID: "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:24:56.978123 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.978100 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-kube-api-access-hzl4p" (OuterVolumeSpecName: "kube-api-access-hzl4p") pod "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" (UID: "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75"). InnerVolumeSpecName "kube-api-access-hzl4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:24:56.978211 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.978124 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" (UID: "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:24:56.978211 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:56.978179 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" (UID: "c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:24:57.076861 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.076824 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-oauth-config\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:57.076861 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.076855 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-config\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:57.076861 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.076869 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzl4p\" (UniqueName: \"kubernetes.io/projected/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-kube-api-access-hzl4p\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:57.077075 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.076881 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-oauth-serving-cert\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:57.077075 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.076893 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-service-ca\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:57.077075 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.076904 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75-console-serving-cert\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:24:57.618594 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.618553 2563 generic.go:356] "Generic (PLEG): container finished" podID="c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" containerID="fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3" exitCode=0 Jul 21 08:24:57.619011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.618607 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744565fbd-965pc" Jul 21 08:24:57.619011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.618625 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744565fbd-965pc" event={"ID":"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75","Type":"ContainerDied","Data":"fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3"} Jul 21 08:24:57.619011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.618675 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744565fbd-965pc" event={"ID":"c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75","Type":"ContainerDied","Data":"beccab31bc4f510b2510a1928144626020e079241690cdf878fbb4b85926ac00"} Jul 21 08:24:57.619011 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.618701 2563 scope.go:117] "RemoveContainer" containerID="fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3" Jul 21 08:24:57.627438 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.627422 2563 scope.go:117] "RemoveContainer" containerID="fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3" Jul 21 08:24:57.627693 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:24:57.627664 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3\": container with ID starting with fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3 not found: ID does not exist" containerID="fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3" Jul 21 08:24:57.627746 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.627693 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3"} err="failed to get container status \"fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3\": rpc error: code = NotFound desc = could not find container \"fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3\": container with ID starting with fbc435b9d0cbc2c36630af3cd29989674d9d648b3e9e49355947d1a4509c78c3 not found: ID does not exist" Jul 21 08:24:57.638637 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.638614 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744565fbd-965pc"] Jul 21 08:24:57.641865 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:57.641846 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-744565fbd-965pc"] Jul 21 08:24:58.357067 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:24:58.357032 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" path="/var/lib/kubelet/pods/c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75/volumes" Jul 21 08:25:02.322470 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:02.322434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:25:02.324854 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:02.324827 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19a14646-a2c6-4a63-8716-6fb7127293cf-metrics-certs\") pod \"network-metrics-daemon-7dgc9\" (UID: \"19a14646-a2c6-4a63-8716-6fb7127293cf\") " pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:25:02.362495 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:02.362463 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bh656\"" Jul 21 08:25:02.369652 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:02.369621 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dgc9" Jul 21 08:25:02.495920 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:02.495895 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7dgc9"] Jul 21 08:25:02.498322 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:25:02.498301 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a14646_a2c6_4a63_8716_6fb7127293cf.slice/crio-1f481aa64210bd07d17a4c9d05c632079c7742b4f309860f940592d0c2984384 WatchSource:0}: Error finding container 1f481aa64210bd07d17a4c9d05c632079c7742b4f309860f940592d0c2984384: Status 404 returned error can't find the container with id 1f481aa64210bd07d17a4c9d05c632079c7742b4f309860f940592d0c2984384 Jul 21 08:25:02.640795 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:02.640759 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7dgc9" event={"ID":"19a14646-a2c6-4a63-8716-6fb7127293cf","Type":"ContainerStarted","Data":"1f481aa64210bd07d17a4c9d05c632079c7742b4f309860f940592d0c2984384"} Jul 21 08:25:04.648810 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:04.648778 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7dgc9" event={"ID":"19a14646-a2c6-4a63-8716-6fb7127293cf","Type":"ContainerStarted","Data":"c2319c29cebd68ed0c11eb2b4a69d2a9d8aaf9c085af609917fb54336baaa6ac"} Jul 21 08:25:04.649194 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:04.648829 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7dgc9" event={"ID":"19a14646-a2c6-4a63-8716-6fb7127293cf","Type":"ContainerStarted","Data":"aa24266fe3ea67319d21076459e2491227c9688f11e97882ff0bf75a794a1bb1"} Jul 21 08:25:04.666796 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:04.666747 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7dgc9" podStartSLOduration=374.355643334 podStartE2EDuration="6m15.666732033s" totalImagesPullingTime="1.311088699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:25:04.665954369 +0000 UTC m=+376.744992699" watchObservedRunningTime="2026-07-21 08:25:04.666732033 +0000 UTC m=+376.745770363" Jul 21 08:25:04.753437 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:04.753400 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:04.848564 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:04.848526 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:11.424622 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:11.424585 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:11.524588 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:11.524558 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:13.277459 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:13.277427 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:13.328786 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:13.328752 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:19.955190 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:19.955155 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75c5497bc7-kz8pd"] Jul 21 08:25:19.955610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:19.955458 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c926ee-ff20-44a2-84da-31998e65e2fa" containerName="registry" Jul 21 08:25:19.955610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:19.955469 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c926ee-ff20-44a2-84da-31998e65e2fa" containerName="registry" Jul 21 08:25:19.955610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:19.955477 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" containerName="console" Jul 21 08:25:19.955610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:19.955483 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" containerName="console" Jul 21 08:25:19.955610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:19.955545 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24a4c8f-749e-4b6b-bb5d-cc8c0648fc75" containerName="console" Jul 21 08:25:19.955610 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:19.955552 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c926ee-ff20-44a2-84da-31998e65e2fa" containerName="registry" Jul 21 08:25:19.958521 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:19.958498 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:19.968630 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:19.968604 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c5497bc7-kz8pd"] Jul 21 08:25:20.007554 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.007511 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:20.077971 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.077932 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-service-ca\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.077971 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.077966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-config\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.078176 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.077983 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-trusted-ca-bundle\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.078176 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.078072 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mf6h\" (UniqueName: \"kubernetes.io/projected/b809968f-96c9-4791-acc9-ba5477e2e5a0-kube-api-access-8mf6h\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.078176 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.078130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-oauth-config\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.078176 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.078149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-oauth-serving-cert\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.078299 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.078182 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-serving-cert\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.100364 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.100333 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:20.178927 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.178882 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mf6h\" (UniqueName: \"kubernetes.io/projected/b809968f-96c9-4791-acc9-ba5477e2e5a0-kube-api-access-8mf6h\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.179127 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.178961 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-oauth-config\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.179127 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.178988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-oauth-serving-cert\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.179127 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.179015 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-serving-cert\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.179127 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.179079 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-service-ca\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.179127 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.179101 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-config\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.179127 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.179122 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-trusted-ca-bundle\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.179860 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.179834 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-oauth-serving-cert\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.180037 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.179894 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-config\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.180037 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.179894 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-service-ca\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.180037 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.180036 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-trusted-ca-bundle\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.181685 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.181664 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-serving-cert\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.181770 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.181727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-oauth-config\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.186711 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.186688 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mf6h\" (UniqueName: \"kubernetes.io/projected/b809968f-96c9-4791-acc9-ba5477e2e5a0-kube-api-access-8mf6h\") pod \"console-75c5497bc7-kz8pd\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.268305 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.268221 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:20.399367 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.399345 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c5497bc7-kz8pd"] Jul 21 08:25:20.402047 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:25:20.402024 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb809968f_96c9_4791_acc9_ba5477e2e5a0.slice/crio-0fc871e563b4ff051befaa4de071af016a3db1d7da80f0a92e6779f15b641ae4 WatchSource:0}: Error finding container 0fc871e563b4ff051befaa4de071af016a3db1d7da80f0a92e6779f15b641ae4: Status 404 returned error can't find the container with id 0fc871e563b4ff051befaa4de071af016a3db1d7da80f0a92e6779f15b641ae4 Jul 21 08:25:20.709272 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.709234 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c5497bc7-kz8pd" event={"ID":"b809968f-96c9-4791-acc9-ba5477e2e5a0","Type":"ContainerStarted","Data":"904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a"} Jul 21 08:25:20.709476 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.709314 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c5497bc7-kz8pd" event={"ID":"b809968f-96c9-4791-acc9-ba5477e2e5a0","Type":"ContainerStarted","Data":"0fc871e563b4ff051befaa4de071af016a3db1d7da80f0a92e6779f15b641ae4"} Jul 21 08:25:20.727266 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:20.727216 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75c5497bc7-kz8pd" podStartSLOduration=1.727200566 podStartE2EDuration="1.727200566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:25:20.725928204 +0000 UTC m=+392.804966548" watchObservedRunningTime="2026-07-21 08:25:20.727200566 +0000 UTC m=+392.806238897" Jul 21 08:25:21.168222 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:21.168176 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:21.226014 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:21.225986 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:30.268954 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:30.268909 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:30.269359 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:30.268980 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:30.273922 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:30.273897 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:30.752656 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:30.752624 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:25:30.794179 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:30.794150 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fb9c5b84-pcl6q"] Jul 21 08:25:30.900209 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:30.900175 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:30.956191 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:30.956160 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:31.063101 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:31.063006 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:52.600595 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:52.600563 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:52.649578 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:52.649545 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:25:55.819948 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:55.819902 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fb9c5b84-pcl6q" podUID="66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" containerName="console" containerID="cri-o://ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698" gracePeriod=15 Jul 21 08:25:56.071921 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.071868 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:25:56.165968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.165926 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-oauth-config\") pod \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " Jul 21 08:25:56.165968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.165983 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-serving-cert\") pod \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " Jul 21 08:25:56.166186 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.166005 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69hc6\" (UniqueName: \"kubernetes.io/projected/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-kube-api-access-69hc6\") pod \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " Jul 21 08:25:56.166186 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.166044 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-trusted-ca-bundle\") pod \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " Jul 21 08:25:56.166186 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.166143 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-service-ca\") pod \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " Jul 21 08:25:56.166186 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.166174 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-config\") pod \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " Jul 21 08:25:56.166408 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.166193 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-oauth-serving-cert\") pod \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\" (UID: \"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb\") " Jul 21 08:25:56.166612 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.166553 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" (UID: "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:56.166741 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.166635 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-config" (OuterVolumeSpecName: "console-config") pod "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" (UID: "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:56.166741 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.166648 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-service-ca" (OuterVolumeSpecName: "service-ca") pod "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" (UID: "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:56.166741 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.166725 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" (UID: "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:56.168350 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.168323 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-kube-api-access-69hc6" (OuterVolumeSpecName: "kube-api-access-69hc6") pod "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" (UID: "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb"). InnerVolumeSpecName "kube-api-access-69hc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:25:56.168689 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.168663 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" (UID: "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:56.168768 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.168720 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" (UID: "66fbe7bf-71ae-404d-a5ec-b788ecdc02eb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:56.267559 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.267518 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-trusted-ca-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:25:56.267559 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.267556 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-service-ca\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:25:56.267559 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.267565 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-config\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:25:56.267559 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.267576 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-oauth-serving-cert\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:25:56.267795 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.267585 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-oauth-config\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:25:56.267795 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.267594 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-console-serving-cert\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:25:56.267795 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.267607 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hc6\" (UniqueName: \"kubernetes.io/projected/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb-kube-api-access-69hc6\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:25:56.849457 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.849419 2563 generic.go:356] "Generic (PLEG): container finished" podID="66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" containerID="ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698" exitCode=0 Jul 21 08:25:56.849914 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.849478 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb9c5b84-pcl6q" Jul 21 08:25:56.849914 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.849491 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb9c5b84-pcl6q" event={"ID":"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb","Type":"ContainerDied","Data":"ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698"} Jul 21 08:25:56.849914 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.849541 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb9c5b84-pcl6q" event={"ID":"66fbe7bf-71ae-404d-a5ec-b788ecdc02eb","Type":"ContainerDied","Data":"cf0c2a5806d22e89c962c93c98ae1ff89fadef28eb83ba31920dd7b4209e860d"} Jul 21 08:25:56.849914 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.849570 2563 scope.go:117] "RemoveContainer" containerID="ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698" Jul 21 08:25:56.857692 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.857672 2563 scope.go:117] "RemoveContainer" containerID="ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698" Jul 21 08:25:56.857942 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:25:56.857919 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698\": container with ID starting with ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698 not found: ID does not exist" containerID="ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698" Jul 21 08:25:56.858005 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.857950 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698"} err="failed to get container status \"ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698\": rpc error: code = NotFound desc = could not find container \"ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698\": container with ID starting with ca5d731237eea42643410657c113981bbf705274bf8dcfd11afdce9495c16698 not found: ID does not exist" Jul 21 08:25:56.864861 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.864840 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fb9c5b84-pcl6q"] Jul 21 08:25:56.868533 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:56.868514 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fb9c5b84-pcl6q"] Jul 21 08:25:58.356573 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:25:58.356546 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" path="/var/lib/kubelet/pods/66fbe7bf-71ae-404d-a5ec-b788ecdc02eb/volumes" Jul 21 08:26:26.635407 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:26.635358 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:26.737197 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:26.737166 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:28.212979 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.212946 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7787b59fcd-dvpcc"] Jul 21 08:26:28.213351 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.213228 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" containerName="console" Jul 21 08:26:28.213351 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.213238 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" containerName="console" Jul 21 08:26:28.213351 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.213293 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="66fbe7bf-71ae-404d-a5ec-b788ecdc02eb" containerName="console" Jul 21 08:26:28.216289 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.216272 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.225980 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.225955 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7787b59fcd-dvpcc"] Jul 21 08:26:28.314016 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.313982 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-oauth-serving-cert\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.314016 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.314026 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64rf\" (UniqueName: \"kubernetes.io/projected/816776a3-d665-4c28-82d1-e6e38c7eb31e-kube-api-access-l64rf\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.314281 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.314060 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-service-ca\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.314281 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.314086 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-trusted-ca-bundle\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.314281 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.314106 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-oauth-config\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.314281 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.314121 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-config\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.314281 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.314138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-serving-cert\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.415349 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.415321 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-oauth-serving-cert\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.415349 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.415359 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64rf\" (UniqueName: \"kubernetes.io/projected/816776a3-d665-4c28-82d1-e6e38c7eb31e-kube-api-access-l64rf\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.415635 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.415431 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-service-ca\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.415635 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.415459 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-trusted-ca-bundle\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.415635 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.415488 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-oauth-config\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.415635 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.415513 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-config\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.415635 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.415553 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-serving-cert\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.416162 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.416131 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-oauth-serving-cert\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.416445 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.416430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-config\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.416620 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.416594 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-service-ca\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.416682 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.416649 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-trusted-ca-bundle\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.418085 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.418065 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-oauth-config\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.418183 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.418166 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-serving-cert\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.422836 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.422819 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64rf\" (UniqueName: \"kubernetes.io/projected/816776a3-d665-4c28-82d1-e6e38c7eb31e-kube-api-access-l64rf\") pod \"console-7787b59fcd-dvpcc\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.483266 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.483192 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:28.524848 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.524816 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:28.655697 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.655668 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7787b59fcd-dvpcc"] Jul 21 08:26:28.659624 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:26:28.659592 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816776a3_d665_4c28_82d1_e6e38c7eb31e.slice/crio-c25eacad994cb833d1e18540819080f2271b04b71ef34b8272ce3e17b28d84ad WatchSource:0}: Error finding container c25eacad994cb833d1e18540819080f2271b04b71ef34b8272ce3e17b28d84ad: Status 404 returned error can't find the container with id c25eacad994cb833d1e18540819080f2271b04b71ef34b8272ce3e17b28d84ad Jul 21 08:26:28.967989 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.967959 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7787b59fcd-dvpcc" event={"ID":"816776a3-d665-4c28-82d1-e6e38c7eb31e","Type":"ContainerStarted","Data":"08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355"} Jul 21 08:26:28.968150 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.968006 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7787b59fcd-dvpcc" event={"ID":"816776a3-d665-4c28-82d1-e6e38c7eb31e","Type":"ContainerStarted","Data":"c25eacad994cb833d1e18540819080f2271b04b71ef34b8272ce3e17b28d84ad"} Jul 21 08:26:28.988372 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:28.988322 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7787b59fcd-dvpcc" podStartSLOduration=0.988307083 podStartE2EDuration="988.307083ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:26:28.987306639 +0000 UTC m=+461.066344981" watchObservedRunningTime="2026-07-21 08:26:28.988307083 +0000 UTC m=+461.067345414" Jul 21 08:26:38.525127 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:38.525089 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:38.525584 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:38.525172 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:38.530064 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:38.530041 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:38.569638 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:38.569602 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:38.627686 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:38.627659 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:38.729196 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:38.729163 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:39.008066 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:39.008041 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:26:39.057283 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:39.057255 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c5497bc7-kz8pd"] Jul 21 08:26:39.164420 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:39.164374 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:39.260261 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:39.260186 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:39.380028 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:39.380002 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:46.368775 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:46.368739 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:26:46.471140 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:26:46.471112 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:27:03.785955 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:03.785916 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:27:04.079281 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.079169 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75c5497bc7-kz8pd" podUID="b809968f-96c9-4791-acc9-ba5477e2e5a0" containerName="console" containerID="cri-o://904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a" gracePeriod=15 Jul 21 08:27:04.325055 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.325029 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:27:04.388214 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388181 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-serving-cert\") pod \"b809968f-96c9-4791-acc9-ba5477e2e5a0\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " Jul 21 08:27:04.388358 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388244 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-oauth-serving-cert\") pod \"b809968f-96c9-4791-acc9-ba5477e2e5a0\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " Jul 21 08:27:04.388358 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388273 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-oauth-config\") pod \"b809968f-96c9-4791-acc9-ba5477e2e5a0\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " Jul 21 08:27:04.388358 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388304 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-service-ca\") pod \"b809968f-96c9-4791-acc9-ba5477e2e5a0\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " Jul 21 08:27:04.388358 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388331 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-trusted-ca-bundle\") pod \"b809968f-96c9-4791-acc9-ba5477e2e5a0\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " Jul 21 08:27:04.388358 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388355 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mf6h\" (UniqueName: \"kubernetes.io/projected/b809968f-96c9-4791-acc9-ba5477e2e5a0-kube-api-access-8mf6h\") pod \"b809968f-96c9-4791-acc9-ba5477e2e5a0\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " Jul 21 08:27:04.388627 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388428 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-config\") pod \"b809968f-96c9-4791-acc9-ba5477e2e5a0\" (UID: \"b809968f-96c9-4791-acc9-ba5477e2e5a0\") " Jul 21 08:27:04.388770 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388737 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b809968f-96c9-4791-acc9-ba5477e2e5a0" (UID: "b809968f-96c9-4791-acc9-ba5477e2e5a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:27:04.388912 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388835 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b809968f-96c9-4791-acc9-ba5477e2e5a0" (UID: "b809968f-96c9-4791-acc9-ba5477e2e5a0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:27:04.388912 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388852 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "b809968f-96c9-4791-acc9-ba5477e2e5a0" (UID: "b809968f-96c9-4791-acc9-ba5477e2e5a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:27:04.389013 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.388933 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-config" (OuterVolumeSpecName: "console-config") pod "b809968f-96c9-4791-acc9-ba5477e2e5a0" (UID: "b809968f-96c9-4791-acc9-ba5477e2e5a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:27:04.390495 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.390459 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b809968f-96c9-4791-acc9-ba5477e2e5a0" (UID: "b809968f-96c9-4791-acc9-ba5477e2e5a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:27:04.391132 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.391107 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b809968f-96c9-4791-acc9-ba5477e2e5a0" (UID: "b809968f-96c9-4791-acc9-ba5477e2e5a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:27:04.391202 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.391126 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b809968f-96c9-4791-acc9-ba5477e2e5a0-kube-api-access-8mf6h" (OuterVolumeSpecName: "kube-api-access-8mf6h") pod "b809968f-96c9-4791-acc9-ba5477e2e5a0" (UID: "b809968f-96c9-4791-acc9-ba5477e2e5a0"). InnerVolumeSpecName "kube-api-access-8mf6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:27:04.489875 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.489840 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-serving-cert\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:27:04.489875 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.489866 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-oauth-serving-cert\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:27:04.489875 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.489876 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-oauth-config\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:27:04.489875 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.489884 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-service-ca\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:27:04.490126 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.489893 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-trusted-ca-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:27:04.490126 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.489901 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mf6h\" (UniqueName: \"kubernetes.io/projected/b809968f-96c9-4791-acc9-ba5477e2e5a0-kube-api-access-8mf6h\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:27:04.490126 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:04.489910 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b809968f-96c9-4791-acc9-ba5477e2e5a0-console-config\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:27:05.100169 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:05.100133 2563 generic.go:356] "Generic (PLEG): container finished" podID="b809968f-96c9-4791-acc9-ba5477e2e5a0" containerID="904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a" exitCode=0 Jul 21 08:27:05.100562 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:05.100177 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c5497bc7-kz8pd" Jul 21 08:27:05.100562 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:05.100227 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c5497bc7-kz8pd" event={"ID":"b809968f-96c9-4791-acc9-ba5477e2e5a0","Type":"ContainerDied","Data":"904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a"} Jul 21 08:27:05.100562 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:05.100278 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c5497bc7-kz8pd" event={"ID":"b809968f-96c9-4791-acc9-ba5477e2e5a0","Type":"ContainerDied","Data":"0fc871e563b4ff051befaa4de071af016a3db1d7da80f0a92e6779f15b641ae4"} Jul 21 08:27:05.100562 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:05.100299 2563 scope.go:117] "RemoveContainer" containerID="904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a" Jul 21 08:27:05.108944 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:05.108931 2563 scope.go:117] "RemoveContainer" containerID="904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a" Jul 21 08:27:05.109196 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:27:05.109178 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a\": container with ID starting with 904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a not found: ID does not exist" containerID="904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a" Jul 21 08:27:05.109258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:05.109204 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a"} err="failed to get container status \"904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a\": rpc error: code = NotFound desc = could not find container \"904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a\": container with ID starting with 904510861c488c1b1ba58c122c7f10251a785a5c74c0a44a0d3833fb5707821a not found: ID does not exist" Jul 21 08:27:05.121283 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:05.121261 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c5497bc7-kz8pd"] Jul 21 08:27:05.125540 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:05.125521 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75c5497bc7-kz8pd"] Jul 21 08:27:06.357101 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:06.357067 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b809968f-96c9-4791-acc9-ba5477e2e5a0" path="/var/lib/kubelet/pods/b809968f-96c9-4791-acc9-ba5477e2e5a0/volumes" Jul 21 08:27:39.628683 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:39.628605 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:27:43.767366 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:43.767334 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:27:43.864636 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:43.864604 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:27:44.644441 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.644408 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b"] Jul 21 08:27:44.644721 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.644710 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b809968f-96c9-4791-acc9-ba5477e2e5a0" containerName="console" Jul 21 08:27:44.644765 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.644723 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b809968f-96c9-4791-acc9-ba5477e2e5a0" containerName="console" Jul 21 08:27:44.644798 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.644788 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b809968f-96c9-4791-acc9-ba5477e2e5a0" containerName="console" Jul 21 08:27:44.647690 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.647667 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.650726 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.650703 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5gmdz\"" Jul 21 08:27:44.650726 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.650716 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 08:27:44.651870 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.651847 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 08:27:44.664626 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.664607 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b"] Jul 21 08:27:44.702012 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.701982 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.702142 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.702031 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tplt9\" (UniqueName: \"kubernetes.io/projected/07607352-695a-4c07-8046-0997ebd22a3c-kube-api-access-tplt9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.702142 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.702051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.802872 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.802841 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.803230 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.802900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tplt9\" (UniqueName: \"kubernetes.io/projected/07607352-695a-4c07-8046-0997ebd22a3c-kube-api-access-tplt9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.803230 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.802927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.803314 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.803226 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.803314 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.803276 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.813049 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.813024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tplt9\" (UniqueName: \"kubernetes.io/projected/07607352-695a-4c07-8046-0997ebd22a3c-kube-api-access-tplt9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:44.957262 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:44.957180 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:27:45.088726 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:45.088694 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b"] Jul 21 08:27:45.092052 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:27:45.092027 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07607352_695a_4c07_8046_0997ebd22a3c.slice/crio-9f6795608dd009269241e5935ff3f5518c37d5f3d19203dfeb67384d29fe8e02 WatchSource:0}: Error finding container 9f6795608dd009269241e5935ff3f5518c37d5f3d19203dfeb67384d29fe8e02: Status 404 returned error can't find the container with id 9f6795608dd009269241e5935ff3f5518c37d5f3d19203dfeb67384d29fe8e02 Jul 21 08:27:45.241082 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:45.241006 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" event={"ID":"07607352-695a-4c07-8046-0997ebd22a3c","Type":"ContainerStarted","Data":"9f6795608dd009269241e5935ff3f5518c37d5f3d19203dfeb67384d29fe8e02"} Jul 21 08:27:53.274521 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:53.274486 2563 generic.go:356] "Generic (PLEG): container finished" podID="07607352-695a-4c07-8046-0997ebd22a3c" containerID="e66390d36492b06ddd3cbf27d485bc5837abd0540b238cc31d96d695ad1e50ff" exitCode=0 Jul 21 08:27:53.274916 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:53.274579 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" event={"ID":"07607352-695a-4c07-8046-0997ebd22a3c","Type":"ContainerDied","Data":"e66390d36492b06ddd3cbf27d485bc5837abd0540b238cc31d96d695ad1e50ff"} Jul 21 08:27:55.284057 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:55.284025 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" event={"ID":"07607352-695a-4c07-8046-0997ebd22a3c","Type":"ContainerStarted","Data":"dded01595321ad08e7ab2f8f467f798e72062746ea6d57b3b2ed6339f7dd5108"} Jul 21 08:27:56.288701 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:56.288667 2563 generic.go:356] "Generic (PLEG): container finished" podID="07607352-695a-4c07-8046-0997ebd22a3c" containerID="dded01595321ad08e7ab2f8f467f798e72062746ea6d57b3b2ed6339f7dd5108" exitCode=0 Jul 21 08:27:56.288701 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:27:56.288707 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" event={"ID":"07607352-695a-4c07-8046-0997ebd22a3c","Type":"ContainerDied","Data":"dded01595321ad08e7ab2f8f467f798e72062746ea6d57b3b2ed6339f7dd5108"} Jul 21 08:28:04.326197 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:04.326110 2563 generic.go:356] "Generic (PLEG): container finished" podID="07607352-695a-4c07-8046-0997ebd22a3c" containerID="5828cb7ec6cd92176dd6a1c8271531a42a39aeeaf5f02f19c7ff63e78cb8e04a" exitCode=0 Jul 21 08:28:04.326558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:04.326196 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" event={"ID":"07607352-695a-4c07-8046-0997ebd22a3c","Type":"ContainerDied","Data":"5828cb7ec6cd92176dd6a1c8271531a42a39aeeaf5f02f19c7ff63e78cb8e04a"} Jul 21 08:28:05.458840 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.458818 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:28:05.562830 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.562793 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-bundle\") pod \"07607352-695a-4c07-8046-0997ebd22a3c\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " Jul 21 08:28:05.562830 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.562836 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-util\") pod \"07607352-695a-4c07-8046-0997ebd22a3c\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " Jul 21 08:28:05.563063 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.562871 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tplt9\" (UniqueName: \"kubernetes.io/projected/07607352-695a-4c07-8046-0997ebd22a3c-kube-api-access-tplt9\") pod \"07607352-695a-4c07-8046-0997ebd22a3c\" (UID: \"07607352-695a-4c07-8046-0997ebd22a3c\") " Jul 21 08:28:05.563537 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.563508 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-bundle" (OuterVolumeSpecName: "bundle") pod "07607352-695a-4c07-8046-0997ebd22a3c" (UID: "07607352-695a-4c07-8046-0997ebd22a3c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:28:05.565101 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.565082 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07607352-695a-4c07-8046-0997ebd22a3c-kube-api-access-tplt9" (OuterVolumeSpecName: "kube-api-access-tplt9") pod "07607352-695a-4c07-8046-0997ebd22a3c" (UID: "07607352-695a-4c07-8046-0997ebd22a3c"). InnerVolumeSpecName "kube-api-access-tplt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:28:05.567015 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.566989 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-util" (OuterVolumeSpecName: "util") pod "07607352-695a-4c07-8046-0997ebd22a3c" (UID: "07607352-695a-4c07-8046-0997ebd22a3c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:28:05.664483 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.664451 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:28:05.664483 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.664480 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07607352-695a-4c07-8046-0997ebd22a3c-util\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:28:05.664483 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:05.664488 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tplt9\" (UniqueName: \"kubernetes.io/projected/07607352-695a-4c07-8046-0997ebd22a3c-kube-api-access-tplt9\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:28:06.335685 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:06.335659 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" Jul 21 08:28:06.335685 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:06.335664 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cdn5b" event={"ID":"07607352-695a-4c07-8046-0997ebd22a3c","Type":"ContainerDied","Data":"9f6795608dd009269241e5935ff3f5518c37d5f3d19203dfeb67384d29fe8e02"} Jul 21 08:28:06.335885 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:06.335717 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f6795608dd009269241e5935ff3f5518c37d5f3d19203dfeb67384d29fe8e02" Jul 21 08:28:10.714338 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:10.714305 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:11.711514 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:11.711487 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:12.675842 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.675809 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh"] Jul 21 08:28:12.676295 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.676076 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07607352-695a-4c07-8046-0997ebd22a3c" containerName="extract" Jul 21 08:28:12.676295 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.676086 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07607352-695a-4c07-8046-0997ebd22a3c" containerName="extract" Jul 21 08:28:12.676295 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.676106 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07607352-695a-4c07-8046-0997ebd22a3c" containerName="util" Jul 21 08:28:12.676295 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.676111 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07607352-695a-4c07-8046-0997ebd22a3c" containerName="util" Jul 21 08:28:12.676295 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.676117 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07607352-695a-4c07-8046-0997ebd22a3c" containerName="pull" Jul 21 08:28:12.676295 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.676122 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07607352-695a-4c07-8046-0997ebd22a3c" containerName="pull" Jul 21 08:28:12.676295 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.676171 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="07607352-695a-4c07-8046-0997ebd22a3c" containerName="extract" Jul 21 08:28:12.679071 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.679056 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" Jul 21 08:28:12.681429 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.681404 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 21 08:28:12.681575 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.681477 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-crmvg\"" Jul 21 08:28:12.681575 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.681512 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 21 08:28:12.692004 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.691982 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh"] Jul 21 08:28:12.718926 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.718897 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bec025a-c5c1-4a1c-9e07-39b39219c620-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzvqh\" (UID: \"1bec025a-c5c1-4a1c-9e07-39b39219c620\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" Jul 21 08:28:12.719063 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.718934 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwkp\" (UniqueName: \"kubernetes.io/projected/1bec025a-c5c1-4a1c-9e07-39b39219c620-kube-api-access-4lwkp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzvqh\" (UID: \"1bec025a-c5c1-4a1c-9e07-39b39219c620\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" Jul 21 08:28:12.819920 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.819871 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bec025a-c5c1-4a1c-9e07-39b39219c620-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzvqh\" (UID: \"1bec025a-c5c1-4a1c-9e07-39b39219c620\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" Jul 21 08:28:12.819920 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.819923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwkp\" (UniqueName: \"kubernetes.io/projected/1bec025a-c5c1-4a1c-9e07-39b39219c620-kube-api-access-4lwkp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzvqh\" (UID: \"1bec025a-c5c1-4a1c-9e07-39b39219c620\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" Jul 21 08:28:12.820290 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.820265 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bec025a-c5c1-4a1c-9e07-39b39219c620-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzvqh\" (UID: \"1bec025a-c5c1-4a1c-9e07-39b39219c620\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" Jul 21 08:28:12.834169 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.834147 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwkp\" (UniqueName: \"kubernetes.io/projected/1bec025a-c5c1-4a1c-9e07-39b39219c620-kube-api-access-4lwkp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hzvqh\" (UID: \"1bec025a-c5c1-4a1c-9e07-39b39219c620\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" Jul 21 08:28:12.988239 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:12.988146 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" Jul 21 08:28:13.120905 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:13.120870 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh"] Jul 21 08:28:13.124091 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:28:13.124058 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bec025a_c5c1_4a1c_9e07_39b39219c620.slice/crio-4490fbecb49d378628b6c75b65e75ec443af1838408a53cd416ebe6db323a22e WatchSource:0}: Error finding container 4490fbecb49d378628b6c75b65e75ec443af1838408a53cd416ebe6db323a22e: Status 404 returned error can't find the container with id 4490fbecb49d378628b6c75b65e75ec443af1838408a53cd416ebe6db323a22e Jul 21 08:28:13.362804 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:13.362721 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" event={"ID":"1bec025a-c5c1-4a1c-9e07-39b39219c620","Type":"ContainerStarted","Data":"4490fbecb49d378628b6c75b65e75ec443af1838408a53cd416ebe6db323a22e"} Jul 21 08:28:14.319665 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:14.315546 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:14.416361 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:14.416328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:15.373437 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:15.373336 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" event={"ID":"1bec025a-c5c1-4a1c-9e07-39b39219c620","Type":"ContainerStarted","Data":"3e0964a568863af62b71314f2ab47a0d9c3726dca15543cc4894e4bead9cdf79"} Jul 21 08:28:15.396347 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:15.396291 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hzvqh" podStartSLOduration=1.48318736 podStartE2EDuration="3.396275049s" totalImagesPullingTime="1.913087689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:28:15.393979475 +0000 UTC m=+567.473017805" watchObservedRunningTime="2026-07-21 08:28:15.396275049 +0000 UTC m=+567.475313380" Jul 21 08:28:15.783769 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:15.783729 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:15.911592 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:15.911563 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:16.151527 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.151498 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-787bp"] Jul 21 08:28:16.154757 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.154739 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:16.157592 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.157570 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 08:28:16.157715 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.157599 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 08:28:16.157715 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.157617 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-r5xjq\"" Jul 21 08:28:16.164815 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.164794 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-787bp"] Jul 21 08:28:16.250708 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.250679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/034e2700-074d-4072-b6ce-b7ff2f140762-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-787bp\" (UID: \"034e2700-074d-4072-b6ce-b7ff2f140762\") " pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:16.250845 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.250718 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phdf\" (UniqueName: \"kubernetes.io/projected/034e2700-074d-4072-b6ce-b7ff2f140762-kube-api-access-2phdf\") pod \"cert-manager-webhook-6486ffd997-787bp\" (UID: \"034e2700-074d-4072-b6ce-b7ff2f140762\") " pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:16.351823 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.351792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/034e2700-074d-4072-b6ce-b7ff2f140762-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-787bp\" (UID: \"034e2700-074d-4072-b6ce-b7ff2f140762\") " pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:16.351992 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.351834 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phdf\" (UniqueName: \"kubernetes.io/projected/034e2700-074d-4072-b6ce-b7ff2f140762-kube-api-access-2phdf\") pod \"cert-manager-webhook-6486ffd997-787bp\" (UID: \"034e2700-074d-4072-b6ce-b7ff2f140762\") " pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:16.361405 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.361362 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/034e2700-074d-4072-b6ce-b7ff2f140762-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-787bp\" (UID: \"034e2700-074d-4072-b6ce-b7ff2f140762\") " pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:16.361547 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.361530 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phdf\" (UniqueName: \"kubernetes.io/projected/034e2700-074d-4072-b6ce-b7ff2f140762-kube-api-access-2phdf\") pod \"cert-manager-webhook-6486ffd997-787bp\" (UID: \"034e2700-074d-4072-b6ce-b7ff2f140762\") " pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:16.473252 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.473173 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:16.606677 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:16.606656 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-787bp"] Jul 21 08:28:17.133612 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.133577 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84"] Jul 21 08:28:17.138052 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.138034 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.140251 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.140233 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5gmdz\"" Jul 21 08:28:17.140455 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.140438 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 08:28:17.141266 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.141250 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 08:28:17.145055 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.145032 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84"] Jul 21 08:28:17.260074 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.260037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7glz\" (UniqueName: \"kubernetes.io/projected/cd90e41a-5f74-4bb2-aee2-b115a289f290-kube-api-access-r7glz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.260247 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.260096 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.260247 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.260132 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.360732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.360701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.360900 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.360745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.360900 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.360789 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7glz\" (UniqueName: \"kubernetes.io/projected/cd90e41a-5f74-4bb2-aee2-b115a289f290-kube-api-access-r7glz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.361077 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.361056 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.361119 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.361081 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.369317 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.369293 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7glz\" (UniqueName: \"kubernetes.io/projected/cd90e41a-5f74-4bb2-aee2-b115a289f290-kube-api-access-r7glz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.382558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.382531 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" event={"ID":"034e2700-074d-4072-b6ce-b7ff2f140762","Type":"ContainerStarted","Data":"16268f6ae3c15e94017b2ec2611dff360360f16754e22bfe7e3ad079458a3610"} Jul 21 08:28:17.447887 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.447854 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:17.579894 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:17.579864 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84"] Jul 21 08:28:17.583145 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:28:17.583121 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd90e41a_5f74_4bb2_aee2_b115a289f290.slice/crio-b2c2377cd391d06466070fb5f176571c7521946b3d55f004421c7f1331d4d61a WatchSource:0}: Error finding container b2c2377cd391d06466070fb5f176571c7521946b3d55f004421c7f1331d4d61a: Status 404 returned error can't find the container with id b2c2377cd391d06466070fb5f176571c7521946b3d55f004421c7f1331d4d61a Jul 21 08:28:18.142069 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:18.142036 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:18.351924 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:18.351884 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:18.389171 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:18.389137 2563 generic.go:356] "Generic (PLEG): container finished" podID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerID="b81fcf6bc36df88891eb0b0831e9affc0d186536d731b3ca4a1635f908c1883d" exitCode=0 Jul 21 08:28:18.389339 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:18.389245 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" event={"ID":"cd90e41a-5f74-4bb2-aee2-b115a289f290","Type":"ContainerDied","Data":"b81fcf6bc36df88891eb0b0831e9affc0d186536d731b3ca4a1635f908c1883d"} Jul 21 08:28:18.389339 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:18.389299 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" event={"ID":"cd90e41a-5f74-4bb2-aee2-b115a289f290","Type":"ContainerStarted","Data":"b2c2377cd391d06466070fb5f176571c7521946b3d55f004421c7f1331d4d61a"} Jul 21 08:28:20.401692 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:20.401654 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" event={"ID":"034e2700-074d-4072-b6ce-b7ff2f140762","Type":"ContainerStarted","Data":"c16e9bd598e3cf584efaf204f13641299ef765cdf533420079c4c632eb13cfb6"} Jul 21 08:28:20.402106 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:20.401815 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:20.419076 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:20.419017 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" podStartSLOduration=1.096000846 podStartE2EDuration="4.418995922s" totalImagesPullingTime="3.322995076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:28:20.41660539 +0000 UTC m=+572.495643721" watchObservedRunningTime="2026-07-21 08:28:20.418995922 +0000 UTC m=+572.498034254" Jul 21 08:28:21.117436 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:21.117399 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:21.169718 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:21.169681 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:22.412953 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.412917 2563 generic.go:356] "Generic (PLEG): container finished" podID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerID="ced267da9327f415de89b0a3e32324c352bf2870819aec0c7f8aaf347d3f9e34" exitCode=0 Jul 21 08:28:22.413346 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.412980 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" event={"ID":"cd90e41a-5f74-4bb2-aee2-b115a289f290","Type":"ContainerDied","Data":"ced267da9327f415de89b0a3e32324c352bf2870819aec0c7f8aaf347d3f9e34"} Jul 21 08:28:22.523819 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.523783 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dhddl"] Jul 21 08:28:22.527604 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.527584 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" Jul 21 08:28:22.529862 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.529843 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gncqk\"" Jul 21 08:28:22.544689 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.544665 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dhddl"] Jul 21 08:28:22.606537 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.606504 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b95edbc-2c1e-4113-b404-772946b890dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dhddl\" (UID: \"7b95edbc-2c1e-4113-b404-772946b890dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" Jul 21 08:28:22.606694 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.606574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvpcg\" (UniqueName: \"kubernetes.io/projected/7b95edbc-2c1e-4113-b404-772946b890dc-kube-api-access-vvpcg\") pod \"cert-manager-cainjector-559cdd8c87-dhddl\" (UID: \"7b95edbc-2c1e-4113-b404-772946b890dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" Jul 21 08:28:22.707964 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.707885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvpcg\" (UniqueName: \"kubernetes.io/projected/7b95edbc-2c1e-4113-b404-772946b890dc-kube-api-access-vvpcg\") pod \"cert-manager-cainjector-559cdd8c87-dhddl\" (UID: \"7b95edbc-2c1e-4113-b404-772946b890dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" Jul 21 08:28:22.707964 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.707959 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b95edbc-2c1e-4113-b404-772946b890dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dhddl\" (UID: \"7b95edbc-2c1e-4113-b404-772946b890dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" Jul 21 08:28:22.715658 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.715628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b95edbc-2c1e-4113-b404-772946b890dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dhddl\" (UID: \"7b95edbc-2c1e-4113-b404-772946b890dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" Jul 21 08:28:22.715778 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.715720 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvpcg\" (UniqueName: \"kubernetes.io/projected/7b95edbc-2c1e-4113-b404-772946b890dc-kube-api-access-vvpcg\") pod \"cert-manager-cainjector-559cdd8c87-dhddl\" (UID: \"7b95edbc-2c1e-4113-b404-772946b890dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" Jul 21 08:28:22.893776 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:22.893740 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" Jul 21 08:28:23.025709 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:23.025684 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dhddl"] Jul 21 08:28:23.028244 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:28:23.028214 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b95edbc_2c1e_4113_b404_772946b890dc.slice/crio-5f7506faece85965fa17617376c2d89d16f37d8583b96c2b8b558186d961f8f6 WatchSource:0}: Error finding container 5f7506faece85965fa17617376c2d89d16f37d8583b96c2b8b558186d961f8f6: Status 404 returned error can't find the container with id 5f7506faece85965fa17617376c2d89d16f37d8583b96c2b8b558186d961f8f6 Jul 21 08:28:23.420283 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:23.420246 2563 generic.go:356] "Generic (PLEG): container finished" podID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerID="b6e6b98baf0f5247fffc31877d58d1165ea28300247e4a50271e6c174ea5e107" exitCode=0 Jul 21 08:28:23.420778 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:23.420322 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" event={"ID":"cd90e41a-5f74-4bb2-aee2-b115a289f290","Type":"ContainerDied","Data":"b6e6b98baf0f5247fffc31877d58d1165ea28300247e4a50271e6c174ea5e107"} Jul 21 08:28:23.421985 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:23.421959 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" event={"ID":"7b95edbc-2c1e-4113-b404-772946b890dc","Type":"ContainerStarted","Data":"b48603dcec353ff0cd9ce314d61d484e48179d8922ed729db9c9ee4330ec4ecd"} Jul 21 08:28:23.422099 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:23.421995 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" event={"ID":"7b95edbc-2c1e-4113-b404-772946b890dc","Type":"ContainerStarted","Data":"5f7506faece85965fa17617376c2d89d16f37d8583b96c2b8b558186d961f8f6"} Jul 21 08:28:23.450687 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:23.450634 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dhddl" podStartSLOduration=1.450620022 podStartE2EDuration="1.450620022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:28:23.44947518 +0000 UTC m=+575.528513511" watchObservedRunningTime="2026-07-21 08:28:23.450620022 +0000 UTC m=+575.529658352" Jul 21 08:28:23.813113 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:23.813031 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:23.912840 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:23.912810 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:24.553368 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.553345 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:24.623320 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.623291 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-util\") pod \"cd90e41a-5f74-4bb2-aee2-b115a289f290\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " Jul 21 08:28:24.623497 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.623343 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-bundle\") pod \"cd90e41a-5f74-4bb2-aee2-b115a289f290\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " Jul 21 08:28:24.623497 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.623367 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7glz\" (UniqueName: \"kubernetes.io/projected/cd90e41a-5f74-4bb2-aee2-b115a289f290-kube-api-access-r7glz\") pod \"cd90e41a-5f74-4bb2-aee2-b115a289f290\" (UID: \"cd90e41a-5f74-4bb2-aee2-b115a289f290\") " Jul 21 08:28:24.623764 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.623741 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-bundle" (OuterVolumeSpecName: "bundle") pod "cd90e41a-5f74-4bb2-aee2-b115a289f290" (UID: "cd90e41a-5f74-4bb2-aee2-b115a289f290"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:28:24.625614 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.625591 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd90e41a-5f74-4bb2-aee2-b115a289f290-kube-api-access-r7glz" (OuterVolumeSpecName: "kube-api-access-r7glz") pod "cd90e41a-5f74-4bb2-aee2-b115a289f290" (UID: "cd90e41a-5f74-4bb2-aee2-b115a289f290"). InnerVolumeSpecName "kube-api-access-r7glz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:28:24.628108 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.628074 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-util" (OuterVolumeSpecName: "util") pod "cd90e41a-5f74-4bb2-aee2-b115a289f290" (UID: "cd90e41a-5f74-4bb2-aee2-b115a289f290"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:28:24.724612 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.724517 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:28:24.724612 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.724550 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7glz\" (UniqueName: \"kubernetes.io/projected/cd90e41a-5f74-4bb2-aee2-b115a289f290-kube-api-access-r7glz\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:28:24.724612 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:24.724560 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd90e41a-5f74-4bb2-aee2-b115a289f290-util\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:28:25.358377 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:25.358341 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:25.432038 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:25.432009 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" event={"ID":"cd90e41a-5f74-4bb2-aee2-b115a289f290","Type":"ContainerDied","Data":"b2c2377cd391d06466070fb5f176571c7521946b3d55f004421c7f1331d4d61a"} Jul 21 08:28:25.432210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:25.432052 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2c2377cd391d06466070fb5f176571c7521946b3d55f004421c7f1331d4d61a" Jul 21 08:28:25.432210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:25.432058 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7t84" Jul 21 08:28:26.408914 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:26.408885 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-787bp" Jul 21 08:28:29.335154 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:29.335117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:29.432064 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:29.432031 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:35.820156 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:35.820122 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:35.920116 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:35.920086 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:37.389765 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.389732 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh"] Jul 21 08:28:37.390157 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.390027 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerName="extract" Jul 21 08:28:37.390157 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.390039 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerName="extract" Jul 21 08:28:37.390157 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.390049 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerName="util" Jul 21 08:28:37.390157 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.390055 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerName="util" Jul 21 08:28:37.390157 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.390067 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerName="pull" Jul 21 08:28:37.390157 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.390072 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerName="pull" Jul 21 08:28:37.390157 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.390117 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd90e41a-5f74-4bb2-aee2-b115a289f290" containerName="extract" Jul 21 08:28:37.394506 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.394491 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.396950 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.396928 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 08:28:37.397062 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.396928 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 08:28:37.397062 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.396929 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5gmdz\"" Jul 21 08:28:37.401841 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.401132 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh"] Jul 21 08:28:37.530600 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.530566 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.530784 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.530649 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52x7\" (UniqueName: \"kubernetes.io/projected/ef369702-6207-4e89-a764-0c7d9e21ee75-kube-api-access-t52x7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.530784 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.530674 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.632009 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.631975 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.632210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.632062 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52x7\" (UniqueName: \"kubernetes.io/projected/ef369702-6207-4e89-a764-0c7d9e21ee75-kube-api-access-t52x7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.632210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.632093 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.632444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.632422 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.632513 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.632449 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.640623 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.640570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52x7\" (UniqueName: \"kubernetes.io/projected/ef369702-6207-4e89-a764-0c7d9e21ee75-kube-api-access-t52x7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.704667 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.704642 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:37.832903 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.832855 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh"] Jul 21 08:28:37.835427 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:28:37.835398 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef369702_6207_4e89_a764_0c7d9e21ee75.slice/crio-d96da4a15a63328833abe04f067a3da040b41e6244710daa0e07af7c139aef64 WatchSource:0}: Error finding container d96da4a15a63328833abe04f067a3da040b41e6244710daa0e07af7c139aef64: Status 404 returned error can't find the container with id d96da4a15a63328833abe04f067a3da040b41e6244710daa0e07af7c139aef64 Jul 21 08:28:37.907893 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:37.907780 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:38.486301 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:38.486263 2563 generic.go:356] "Generic (PLEG): container finished" podID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerID="e3b06e8afbbc192761b611c3b5b631e76ab559e46d27d43b7961e2114245368f" exitCode=0 Jul 21 08:28:38.486703 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:38.486357 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" event={"ID":"ef369702-6207-4e89-a764-0c7d9e21ee75","Type":"ContainerDied","Data":"e3b06e8afbbc192761b611c3b5b631e76ab559e46d27d43b7961e2114245368f"} Jul 21 08:28:38.486703 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:38.486426 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" event={"ID":"ef369702-6207-4e89-a764-0c7d9e21ee75","Type":"ContainerStarted","Data":"d96da4a15a63328833abe04f067a3da040b41e6244710daa0e07af7c139aef64"} Jul 21 08:28:39.492037 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:39.491949 2563 generic.go:356] "Generic (PLEG): container finished" podID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerID="80eea9924bcc3afd161422c0062584051d8dd4507481b5d7616e4ad362ba9ca8" exitCode=0 Jul 21 08:28:39.492420 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:39.492038 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" event={"ID":"ef369702-6207-4e89-a764-0c7d9e21ee75","Type":"ContainerDied","Data":"80eea9924bcc3afd161422c0062584051d8dd4507481b5d7616e4ad362ba9ca8"} Jul 21 08:28:39.679997 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:28:39.679966 2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef369702_6207_4e89_a764_0c7d9e21ee75.slice/crio-conmon-9117f6e4f70a25f75534e19963423c7b7832e8fa302927cd5e8a81bd583db3b6.scope\": RecentStats: unable to find data in memory cache]" Jul 21 08:28:40.498393 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:40.498355 2563 generic.go:356] "Generic (PLEG): container finished" podID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerID="9117f6e4f70a25f75534e19963423c7b7832e8fa302927cd5e8a81bd583db3b6" exitCode=0 Jul 21 08:28:40.498781 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:40.498421 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" event={"ID":"ef369702-6207-4e89-a764-0c7d9e21ee75","Type":"ContainerDied","Data":"9117f6e4f70a25f75534e19963423c7b7832e8fa302927cd5e8a81bd583db3b6"} Jul 21 08:28:41.633721 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.633697 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:41.766441 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.766346 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-util\") pod \"ef369702-6207-4e89-a764-0c7d9e21ee75\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " Jul 21 08:28:41.766441 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.766407 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t52x7\" (UniqueName: \"kubernetes.io/projected/ef369702-6207-4e89-a764-0c7d9e21ee75-kube-api-access-t52x7\") pod \"ef369702-6207-4e89-a764-0c7d9e21ee75\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " Jul 21 08:28:41.766441 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.766431 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-bundle\") pod \"ef369702-6207-4e89-a764-0c7d9e21ee75\" (UID: \"ef369702-6207-4e89-a764-0c7d9e21ee75\") " Jul 21 08:28:41.767179 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.767150 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-bundle" (OuterVolumeSpecName: "bundle") pod "ef369702-6207-4e89-a764-0c7d9e21ee75" (UID: "ef369702-6207-4e89-a764-0c7d9e21ee75"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:28:41.768719 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.768689 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef369702-6207-4e89-a764-0c7d9e21ee75-kube-api-access-t52x7" (OuterVolumeSpecName: "kube-api-access-t52x7") pod "ef369702-6207-4e89-a764-0c7d9e21ee75" (UID: "ef369702-6207-4e89-a764-0c7d9e21ee75"). InnerVolumeSpecName "kube-api-access-t52x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:28:41.772560 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.772527 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-util" (OuterVolumeSpecName: "util") pod "ef369702-6207-4e89-a764-0c7d9e21ee75" (UID: "ef369702-6207-4e89-a764-0c7d9e21ee75"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:28:41.867313 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.867283 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-util\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:28:41.867313 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.867308 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52x7\" (UniqueName: \"kubernetes.io/projected/ef369702-6207-4e89-a764-0c7d9e21ee75-kube-api-access-t52x7\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:28:41.867313 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:41.867319 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef369702-6207-4e89-a764-0c7d9e21ee75-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:28:42.508033 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:42.508004 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" Jul 21 08:28:42.508033 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:42.508020 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhrhmh" event={"ID":"ef369702-6207-4e89-a764-0c7d9e21ee75","Type":"ContainerDied","Data":"d96da4a15a63328833abe04f067a3da040b41e6244710daa0e07af7c139aef64"} Jul 21 08:28:42.508269 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:42.508066 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d96da4a15a63328833abe04f067a3da040b41e6244710daa0e07af7c139aef64" Jul 21 08:28:50.139087 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.139009 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:50.165432 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.165402 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7686776559-xvtl4"] Jul 21 08:28:50.165707 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.165694 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerName="extract" Jul 21 08:28:50.165748 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.165711 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerName="extract" Jul 21 08:28:50.165748 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.165727 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerName="util" Jul 21 08:28:50.165748 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.165734 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerName="util" Jul 21 08:28:50.165748 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.165741 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerName="pull" Jul 21 08:28:50.165748 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.165747 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerName="pull" Jul 21 08:28:50.165937 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.165803 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef369702-6207-4e89-a764-0c7d9e21ee75" containerName="extract" Jul 21 08:28:50.168572 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.168558 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.172203 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.172155 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 08:28:50.172203 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.172187 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 08:28:50.172443 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.172192 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fmjmb\"" Jul 21 08:28:50.172443 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.172237 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 08:28:50.172443 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.172245 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 08:28:50.172443 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.172293 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 08:28:50.177526 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.177507 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7686776559-xvtl4"] Jul 21 08:28:50.212421 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.212364 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:50.234938 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.234909 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-metrics-cert\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.235072 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.234988 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-cert\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.235072 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.235030 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfc2\" (UniqueName: \"kubernetes.io/projected/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-kube-api-access-6xfc2\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.235072 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.235057 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-manager-config\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.336199 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.336159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-metrics-cert\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.336373 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.336219 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-cert\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.336373 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.336237 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfc2\" (UniqueName: \"kubernetes.io/projected/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-kube-api-access-6xfc2\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.336373 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.336255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-manager-config\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.336970 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.336940 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-manager-config\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.338839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.338820 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-cert\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.338919 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.338890 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-metrics-cert\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.347732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.347708 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfc2\" (UniqueName: \"kubernetes.io/projected/612a5640-1975-4c4a-b9e5-9722e4c2d3c3-kube-api-access-6xfc2\") pod \"lws-controller-manager-7686776559-xvtl4\" (UID: \"612a5640-1975-4c4a-b9e5-9722e4c2d3c3\") " pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.478623 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.478527 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:50.607666 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:50.607641 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7686776559-xvtl4"] Jul 21 08:28:50.609926 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:28:50.609898 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612a5640_1975_4c4a_b9e5_9722e4c2d3c3.slice/crio-34d250aa668cdbced727c35fb93f7fdd6a11a90c09d31b6f86ceef8afda155e1 WatchSource:0}: Error finding container 34d250aa668cdbced727c35fb93f7fdd6a11a90c09d31b6f86ceef8afda155e1: Status 404 returned error can't find the container with id 34d250aa668cdbced727c35fb93f7fdd6a11a90c09d31b6f86ceef8afda155e1 Jul 21 08:28:51.549507 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:51.549465 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" event={"ID":"612a5640-1975-4c4a-b9e5-9722e4c2d3c3","Type":"ContainerStarted","Data":"34d250aa668cdbced727c35fb93f7fdd6a11a90c09d31b6f86ceef8afda155e1"} Jul 21 08:28:52.555845 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:52.555753 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" event={"ID":"612a5640-1975-4c4a-b9e5-9722e4c2d3c3","Type":"ContainerStarted","Data":"a2b908824207651da0776b342e7874fd5eff5790819377dfdbd8d7dd3f17de79"} Jul 21 08:28:52.556204 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:52.555893 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:28:52.578375 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:52.578327 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" podStartSLOduration=0.971855054 podStartE2EDuration="2.578311626s" totalImagesPullingTime="1.606456572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:28:52.577472059 +0000 UTC m=+604.656510392" watchObservedRunningTime="2026-07-21 08:28:52.578311626 +0000 UTC m=+604.657349957" Jul 21 08:28:53.527950 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:53.527919 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:54.761748 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.761714 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk"] Jul 21 08:28:54.765110 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.765094 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.769636 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.769617 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 08:28:54.769892 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.769877 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 08:28:54.770504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.770480 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-568nx\"" Jul 21 08:28:54.770604 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.770565 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 08:28:54.770686 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.770671 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 08:28:54.780233 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.780213 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk"] Jul 21 08:28:54.878583 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.878538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s5vj\" (UniqueName: \"kubernetes.io/projected/98373695-9159-4926-9b85-c37aaeb4bc3c-kube-api-access-6s5vj\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.878741 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.878611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/98373695-9159-4926-9b85-c37aaeb4bc3c-manifests\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.878741 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.878669 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/98373695-9159-4926-9b85-c37aaeb4bc3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.878741 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.878706 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98373695-9159-4926-9b85-c37aaeb4bc3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.878741 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.878730 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98373695-9159-4926-9b85-c37aaeb4bc3c-tmp\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.979967 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.979930 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98373695-9159-4926-9b85-c37aaeb4bc3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.979967 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.979968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98373695-9159-4926-9b85-c37aaeb4bc3c-tmp\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.980198 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.980021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s5vj\" (UniqueName: \"kubernetes.io/projected/98373695-9159-4926-9b85-c37aaeb4bc3c-kube-api-access-6s5vj\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.980198 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.980078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/98373695-9159-4926-9b85-c37aaeb4bc3c-manifests\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.980198 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.980117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/98373695-9159-4926-9b85-c37aaeb4bc3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.980612 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.980588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/98373695-9159-4926-9b85-c37aaeb4bc3c-manifests\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.982548 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.982524 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98373695-9159-4926-9b85-c37aaeb4bc3c-tmp\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.982713 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.982694 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/98373695-9159-4926-9b85-c37aaeb4bc3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.982776 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.982702 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98373695-9159-4926-9b85-c37aaeb4bc3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:54.987782 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:54.987765 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s5vj\" (UniqueName: \"kubernetes.io/projected/98373695-9159-4926-9b85-c37aaeb4bc3c-kube-api-access-6s5vj\") pod \"opendatahub-operator-controller-manager-787895fdf5-brfgk\" (UID: \"98373695-9159-4926-9b85-c37aaeb4bc3c\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:55.075585 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:55.075489 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:28:55.214106 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:55.214077 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk"] Jul 21 08:28:55.217367 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:28:55.217343 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98373695_9159_4926_9b85_c37aaeb4bc3c.slice/crio-f2454afaa0d231d2f502b2ccf29881fe05f27fe89bc44e3052cf3e908a433e13 WatchSource:0}: Error finding container f2454afaa0d231d2f502b2ccf29881fe05f27fe89bc44e3052cf3e908a433e13: Status 404 returned error can't find the container with id f2454afaa0d231d2f502b2ccf29881fe05f27fe89bc44e3052cf3e908a433e13 Jul 21 08:28:55.219805 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:55.219788 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:28:55.243879 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:55.243855 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:55.380150 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:55.380062 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:55.478194 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:55.478164 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:28:55.570191 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:55.570158 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" event={"ID":"98373695-9159-4926-9b85-c37aaeb4bc3c","Type":"ContainerStarted","Data":"f2454afaa0d231d2f502b2ccf29881fe05f27fe89bc44e3052cf3e908a433e13"} Jul 21 08:28:58.591506 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:58.591465 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" event={"ID":"98373695-9159-4926-9b85-c37aaeb4bc3c","Type":"ContainerStarted","Data":"9b8add49d5ee11686df592c8005d3dc19e101c75db13f886fc43b67d2ecae520"} Jul 21 08:28:59.598165 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:59.598126 2563 generic.go:356] "Generic (PLEG): container finished" podID="98373695-9159-4926-9b85-c37aaeb4bc3c" containerID="9b8add49d5ee11686df592c8005d3dc19e101c75db13f886fc43b67d2ecae520" exitCode=0 Jul 21 08:28:59.598591 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:28:59.598210 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" event={"ID":"98373695-9159-4926-9b85-c37aaeb4bc3c","Type":"ContainerDied","Data":"9b8add49d5ee11686df592c8005d3dc19e101c75db13f886fc43b67d2ecae520"} Jul 21 08:29:00.605996 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:00.605958 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" event={"ID":"98373695-9159-4926-9b85-c37aaeb4bc3c","Type":"ContainerStarted","Data":"42f488380a68aaa8930eb3321b4e1dbcb46e647316ab100f420dffcbe101eaf8"} Jul 21 08:29:00.606514 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:00.606100 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:29:00.626872 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:00.626828 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" podStartSLOduration=2.040357125 podStartE2EDuration="6.626814028s" totalImagesPullingTime="3.552469998s" totalInitContainerRuntime="1.033986905s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:28:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:00.62544525 +0000 UTC m=+612.704483582" watchObservedRunningTime="2026-07-21 08:29:00.626814028 +0000 UTC m=+612.705852359" Jul 21 08:29:03.561853 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:03.561820 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7686776559-xvtl4" Jul 21 08:29:09.537438 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:09.537362 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:09.590926 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:09.590896 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:09.966137 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:09.966087 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:10.021497 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:10.021462 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:11.613763 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:11.613729 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-brfgk" Jul 21 08:29:13.770148 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:13.770109 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:13.863996 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:13.863966 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:14.179362 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.179326 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:14.300189 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.300160 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq"] Jul 21 08:29:14.372427 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.372368 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:14.376102 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.376079 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sv5ns\"" Jul 21 08:29:14.376244 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.376124 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 08:29:14.376525 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.376507 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 08:29:14.378210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.378189 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq"] Jul 21 08:29:14.455411 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.455317 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3050721d-8325-4e44-a7f5-376af01c2765-tls-certs\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:14.455411 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.455402 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3050721d-8325-4e44-a7f5-376af01c2765-tmp\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:14.455602 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.455461 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g95xd\" (UniqueName: \"kubernetes.io/projected/3050721d-8325-4e44-a7f5-376af01c2765-kube-api-access-g95xd\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:14.556927 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.556891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3050721d-8325-4e44-a7f5-376af01c2765-tmp\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:14.557126 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.556964 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g95xd\" (UniqueName: \"kubernetes.io/projected/3050721d-8325-4e44-a7f5-376af01c2765-kube-api-access-g95xd\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:14.557126 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.557005 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3050721d-8325-4e44-a7f5-376af01c2765-tls-certs\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:14.557255 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:29:14.557150 2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 21 08:29:14.557255 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:29:14.557227 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3050721d-8325-4e44-a7f5-376af01c2765-tls-certs podName:3050721d-8325-4e44-a7f5-376af01c2765 nodeName:}" failed. No retries permitted until 2026-07-21 08:29:15.057206834 +0000 UTC m=+627.136245145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3050721d-8325-4e44-a7f5-376af01c2765-tls-certs") pod "kube-auth-proxy-7d57bc4cd9-bw7rq" (UID: "3050721d-8325-4e44-a7f5-376af01c2765") : secret "kube-auth-proxy-tls" not found Jul 21 08:29:14.560352 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.560319 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3050721d-8325-4e44-a7f5-376af01c2765-tmp\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:14.575222 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.575196 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g95xd\" (UniqueName: \"kubernetes.io/projected/3050721d-8325-4e44-a7f5-376af01c2765-kube-api-access-g95xd\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:14.588996 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.588965 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:14.681618 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.681576 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:14.828963 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:14.828880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:15.061876 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:15.061839 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3050721d-8325-4e44-a7f5-376af01c2765-tls-certs\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:15.064508 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:15.064473 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3050721d-8325-4e44-a7f5-376af01c2765-tls-certs\") pod \"kube-auth-proxy-7d57bc4cd9-bw7rq\" (UID: \"3050721d-8325-4e44-a7f5-376af01c2765\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:15.247566 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:15.247529 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:15.282458 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:15.282422 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" Jul 21 08:29:15.427213 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:15.427073 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq"] Jul 21 08:29:15.608141 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:15.608057 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:15.674084 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:15.674047 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" event={"ID":"3050721d-8325-4e44-a7f5-376af01c2765","Type":"ContainerStarted","Data":"200831623a96c3e3aa2573f73e4fdad88d99d4c7b9657841a94c670fdda0b641"} Jul 21 08:29:15.674917 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:15.674892 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:16.556200 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:16.556163 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:18.692560 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:18.692525 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" event={"ID":"3050721d-8325-4e44-a7f5-376af01c2765","Type":"ContainerStarted","Data":"7d52075a0abb512a49901883e241070ef11bc274c89c2898550330ef49ebb409"} Jul 21 08:29:18.707395 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:18.707342 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-bw7rq" podStartSLOduration=1.560104412 podStartE2EDuration="4.707322162s" totalImagesPullingTime="3.14721775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:18.706425422 +0000 UTC m=+630.785463754" watchObservedRunningTime="2026-07-21 08:29:18.707322162 +0000 UTC m=+630.786360495" Jul 21 08:29:19.175699 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:19.175668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:19.229674 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:19.229641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:21.034941 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:21.034903 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:21.146546 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:21.146515 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:21.754189 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:21.754161 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:21.835883 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:21.835849 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:22.437165 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:22.437134 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:22.783771 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:22.783679 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:23.846057 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:23.846020 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:24.513477 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:24.513449 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:24.639022 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:24.638994 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:25.065348 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:25.065314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:25.281253 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:25.281226 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:25.809788 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:25.809751 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:25.921587 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:25.921548 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:26.014787 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:26.014758 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:28.468248 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:28.468221 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:31.353034 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:31.352997 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:31.453193 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:31.453146 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:29:31.511029 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:29:31.511001 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:15.226506 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:15.226472 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:15.320358 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:15.320330 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:15.417696 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:15.417668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:16.209444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:16.209411 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:16.352516 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:16.352484 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:30.755710 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.755678 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn"] Jul 21 08:30:30.759041 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.759020 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:30.761456 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.761433 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 08:30:30.762326 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.762307 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 08:30:30.762421 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.762329 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cc662\"" Jul 21 08:30:30.767915 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.767893 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn"] Jul 21 08:30:30.810802 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.810773 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkgsb\" (UniqueName: \"kubernetes.io/projected/5d16c9bc-9cee-4761-b59d-133d91538fbc-kube-api-access-zkgsb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:30.810802 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.810804 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:30.810999 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.810909 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:30.912169 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.912134 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:30.912344 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.912181 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkgsb\" (UniqueName: \"kubernetes.io/projected/5d16c9bc-9cee-4761-b59d-133d91538fbc-kube-api-access-zkgsb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:30.912344 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.912303 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:30.912582 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.912563 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:30.912646 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.912631 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:30.921098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:30.921069 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkgsb\" (UniqueName: \"kubernetes.io/projected/5d16c9bc-9cee-4761-b59d-133d91538fbc-kube-api-access-zkgsb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:31.068319 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.068237 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:31.200683 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.200653 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn"] Jul 21 08:30:31.203900 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:30:31.203867 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d16c9bc_9cee_4761_b59d_133d91538fbc.slice/crio-30c1e8badd1b8b5d1630c6341270ccf4863da3f818c47a8dbd4be929c444f689 WatchSource:0}: Error finding container 30c1e8badd1b8b5d1630c6341270ccf4863da3f818c47a8dbd4be929c444f689: Status 404 returned error can't find the container with id 30c1e8badd1b8b5d1630c6341270ccf4863da3f818c47a8dbd4be929c444f689 Jul 21 08:30:31.561318 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.561285 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c"] Jul 21 08:30:31.564707 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.564689 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.573058 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.573035 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c"] Jul 21 08:30:31.617584 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.617554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.617584 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.617588 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kw27\" (UniqueName: \"kubernetes.io/projected/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-kube-api-access-6kw27\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.617808 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.617617 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.718479 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.718444 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.718662 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.718483 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kw27\" (UniqueName: \"kubernetes.io/projected/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-kube-api-access-6kw27\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.718662 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.718525 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.718912 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.718890 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.718912 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.718905 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.727641 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.727615 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kw27\" (UniqueName: \"kubernetes.io/projected/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-kube-api-access-6kw27\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:31.873971 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:31.873897 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:32.006120 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.006098 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c"] Jul 21 08:30:32.008044 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:30:32.008017 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85e6820_7dbb_4da0_8fbd_809f3b06bc61.slice/crio-b5532c9af38e7b883dc719a3d1eb0d63c9433a811316ec1adc523f38cc52f8f6 WatchSource:0}: Error finding container b5532c9af38e7b883dc719a3d1eb0d63c9433a811316ec1adc523f38cc52f8f6: Status 404 returned error can't find the container with id b5532c9af38e7b883dc719a3d1eb0d63c9433a811316ec1adc523f38cc52f8f6 Jul 21 08:30:32.030464 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.030437 2563 generic.go:356] "Generic (PLEG): container finished" podID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerID="ecb349f89db33b23048ddedfe0af0c881f8f0cf8a52fbce265939e1e7d99db45" exitCode=0 Jul 21 08:30:32.030575 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.030522 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" event={"ID":"5d16c9bc-9cee-4761-b59d-133d91538fbc","Type":"ContainerDied","Data":"ecb349f89db33b23048ddedfe0af0c881f8f0cf8a52fbce265939e1e7d99db45"} Jul 21 08:30:32.030636 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.030574 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" event={"ID":"5d16c9bc-9cee-4761-b59d-133d91538fbc","Type":"ContainerStarted","Data":"30c1e8badd1b8b5d1630c6341270ccf4863da3f818c47a8dbd4be929c444f689"} Jul 21 08:30:32.031882 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.031857 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" event={"ID":"b85e6820-7dbb-4da0-8fbd-809f3b06bc61","Type":"ContainerStarted","Data":"b5532c9af38e7b883dc719a3d1eb0d63c9433a811316ec1adc523f38cc52f8f6"} Jul 21 08:30:32.157227 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.157193 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp"] Jul 21 08:30:32.160607 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.160589 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.169940 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.169915 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp"] Jul 21 08:30:32.223134 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.223103 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzv8\" (UniqueName: \"kubernetes.io/projected/6f46db13-b76f-412d-bc4b-1d4477609776-kube-api-access-5wzv8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.223287 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.223170 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.223287 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.223216 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.324086 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.324050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzv8\" (UniqueName: \"kubernetes.io/projected/6f46db13-b76f-412d-bc4b-1d4477609776-kube-api-access-5wzv8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.324258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.324128 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.324258 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.324161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.324547 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.324531 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.324593 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.324570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.332797 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.332774 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzv8\" (UniqueName: \"kubernetes.io/projected/6f46db13-b76f-412d-bc4b-1d4477609776-kube-api-access-5wzv8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.470020 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.469935 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:32.562516 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.562488 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n"] Jul 21 08:30:32.568071 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.568051 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.579617 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.579589 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n"] Jul 21 08:30:32.608277 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.608251 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp"] Jul 21 08:30:32.628047 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.628013 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.628192 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.628067 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.628231 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.628215 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp44z\" (UniqueName: \"kubernetes.io/projected/cba30917-00d2-477b-8a8d-9428d179dc34-kube-api-access-wp44z\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.720850 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:30:32.720783 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f46db13_b76f_412d_bc4b_1d4477609776.slice/crio-151379b2f6e456063962054c19377c641988ae1f0ec0785ceb70cca14f3827e2 WatchSource:0}: Error finding container 151379b2f6e456063962054c19377c641988ae1f0ec0785ceb70cca14f3827e2: Status 404 returned error can't find the container with id 151379b2f6e456063962054c19377c641988ae1f0ec0785ceb70cca14f3827e2 Jul 21 08:30:32.729077 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.729056 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.729154 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.729124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp44z\" (UniqueName: \"kubernetes.io/projected/cba30917-00d2-477b-8a8d-9428d179dc34-kube-api-access-wp44z\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.729219 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.729156 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.729450 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.729429 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.729450 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.729441 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.737932 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.737904 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp44z\" (UniqueName: \"kubernetes.io/projected/cba30917-00d2-477b-8a8d-9428d179dc34-kube-api-access-wp44z\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:32.880071 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:32.880039 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:33.009645 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:33.009575 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n"] Jul 21 08:30:33.012880 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:30:33.012847 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba30917_00d2_477b_8a8d_9428d179dc34.slice/crio-04c0b0f912f7d87f654378f263654fe41e6eee0fff6898402780c0c3711ae733 WatchSource:0}: Error finding container 04c0b0f912f7d87f654378f263654fe41e6eee0fff6898402780c0c3711ae733: Status 404 returned error can't find the container with id 04c0b0f912f7d87f654378f263654fe41e6eee0fff6898402780c0c3711ae733 Jul 21 08:30:33.037887 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:33.037858 2563 generic.go:356] "Generic (PLEG): container finished" podID="6f46db13-b76f-412d-bc4b-1d4477609776" containerID="4967cdd1efff1aa9a42005b053234c24c84ec26522ad3cc8d977e0580b63579a" exitCode=0 Jul 21 08:30:33.038044 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:33.037927 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" event={"ID":"6f46db13-b76f-412d-bc4b-1d4477609776","Type":"ContainerDied","Data":"4967cdd1efff1aa9a42005b053234c24c84ec26522ad3cc8d977e0580b63579a"} Jul 21 08:30:33.038044 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:33.037976 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" event={"ID":"6f46db13-b76f-412d-bc4b-1d4477609776","Type":"ContainerStarted","Data":"151379b2f6e456063962054c19377c641988ae1f0ec0785ceb70cca14f3827e2"} Jul 21 08:30:33.039763 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:33.039739 2563 generic.go:356] "Generic (PLEG): container finished" podID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerID="8dce4e85a388ac2bacbd6649d9b9b77bd01591777e9f8d908c5228900e430e42" exitCode=0 Jul 21 08:30:33.039856 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:33.039802 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" event={"ID":"5d16c9bc-9cee-4761-b59d-133d91538fbc","Type":"ContainerDied","Data":"8dce4e85a388ac2bacbd6649d9b9b77bd01591777e9f8d908c5228900e430e42"} Jul 21 08:30:33.041433 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:33.041362 2563 generic.go:356] "Generic (PLEG): container finished" podID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerID="b7451e9f2493ae9f680bd9a99de8a63d58e9ead8c2e384b4cc59d2fb04ea739b" exitCode=0 Jul 21 08:30:33.041495 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:33.041421 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" event={"ID":"b85e6820-7dbb-4da0-8fbd-809f3b06bc61","Type":"ContainerDied","Data":"b7451e9f2493ae9f680bd9a99de8a63d58e9ead8c2e384b4cc59d2fb04ea739b"} Jul 21 08:30:33.042726 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:33.042709 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" event={"ID":"cba30917-00d2-477b-8a8d-9428d179dc34","Type":"ContainerStarted","Data":"04c0b0f912f7d87f654378f263654fe41e6eee0fff6898402780c0c3711ae733"} Jul 21 08:30:34.050036 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:34.050004 2563 generic.go:356] "Generic (PLEG): container finished" podID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerID="ad527d67a10ece6e399b21f50a85d8a960fc7706913f258f1d6a3ab568c76391" exitCode=0 Jul 21 08:30:34.050510 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:34.050092 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" event={"ID":"b85e6820-7dbb-4da0-8fbd-809f3b06bc61","Type":"ContainerDied","Data":"ad527d67a10ece6e399b21f50a85d8a960fc7706913f258f1d6a3ab568c76391"} Jul 21 08:30:34.051931 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:34.051616 2563 generic.go:356] "Generic (PLEG): container finished" podID="cba30917-00d2-477b-8a8d-9428d179dc34" containerID="3a967ea2f28de98f6b554b0a48bd1efd3ae562327d31919eb5886e45e6cc39d9" exitCode=0 Jul 21 08:30:34.051931 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:34.051637 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" event={"ID":"cba30917-00d2-477b-8a8d-9428d179dc34","Type":"ContainerDied","Data":"3a967ea2f28de98f6b554b0a48bd1efd3ae562327d31919eb5886e45e6cc39d9"} Jul 21 08:30:34.053715 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:34.053695 2563 generic.go:356] "Generic (PLEG): container finished" podID="6f46db13-b76f-412d-bc4b-1d4477609776" containerID="5b57294d52b102760385b4bfda1dc0caf41c1853da9edc1f70bbf0c062fde41d" exitCode=0 Jul 21 08:30:34.053812 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:34.053739 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" event={"ID":"6f46db13-b76f-412d-bc4b-1d4477609776","Type":"ContainerDied","Data":"5b57294d52b102760385b4bfda1dc0caf41c1853da9edc1f70bbf0c062fde41d"} Jul 21 08:30:34.056249 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:34.056229 2563 generic.go:356] "Generic (PLEG): container finished" podID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerID="e9137ccee245a559744e4a2bbd5f2930d73c2f7dd672629462428278a98ddce1" exitCode=0 Jul 21 08:30:34.056337 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:34.056289 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" event={"ID":"5d16c9bc-9cee-4761-b59d-133d91538fbc","Type":"ContainerDied","Data":"e9137ccee245a559744e4a2bbd5f2930d73c2f7dd672629462428278a98ddce1"} Jul 21 08:30:35.064760 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.064676 2563 generic.go:356] "Generic (PLEG): container finished" podID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerID="31b80087fe902ca2f07e432becbef35d5c856bb468eb3b209b3e1ad8ec22dc83" exitCode=0 Jul 21 08:30:35.065153 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.064768 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" event={"ID":"b85e6820-7dbb-4da0-8fbd-809f3b06bc61","Type":"ContainerDied","Data":"31b80087fe902ca2f07e432becbef35d5c856bb468eb3b209b3e1ad8ec22dc83"} Jul 21 08:30:35.066474 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.066452 2563 generic.go:356] "Generic (PLEG): container finished" podID="cba30917-00d2-477b-8a8d-9428d179dc34" containerID="674c39dde7a290f11c920d9438742449486b74c5f54830d4622555f17bb82d53" exitCode=0 Jul 21 08:30:35.066577 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.066533 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" event={"ID":"cba30917-00d2-477b-8a8d-9428d179dc34","Type":"ContainerDied","Data":"674c39dde7a290f11c920d9438742449486b74c5f54830d4622555f17bb82d53"} Jul 21 08:30:35.068768 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.068742 2563 generic.go:356] "Generic (PLEG): container finished" podID="6f46db13-b76f-412d-bc4b-1d4477609776" containerID="77f83dc61515cd7a98dbc680b3146108627e89fefc4bf9768b26609f5e4fe110" exitCode=0 Jul 21 08:30:35.068876 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.068818 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" event={"ID":"6f46db13-b76f-412d-bc4b-1d4477609776","Type":"ContainerDied","Data":"77f83dc61515cd7a98dbc680b3146108627e89fefc4bf9768b26609f5e4fe110"} Jul 21 08:30:35.209512 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.209479 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:35.252703 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.252669 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkgsb\" (UniqueName: \"kubernetes.io/projected/5d16c9bc-9cee-4761-b59d-133d91538fbc-kube-api-access-zkgsb\") pod \"5d16c9bc-9cee-4761-b59d-133d91538fbc\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " Jul 21 08:30:35.252860 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.252720 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-util\") pod \"5d16c9bc-9cee-4761-b59d-133d91538fbc\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " Jul 21 08:30:35.252940 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.252918 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-bundle\") pod \"5d16c9bc-9cee-4761-b59d-133d91538fbc\" (UID: \"5d16c9bc-9cee-4761-b59d-133d91538fbc\") " Jul 21 08:30:35.253577 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.253551 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-bundle" (OuterVolumeSpecName: "bundle") pod "5d16c9bc-9cee-4761-b59d-133d91538fbc" (UID: "5d16c9bc-9cee-4761-b59d-133d91538fbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:30:35.255210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.255176 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d16c9bc-9cee-4761-b59d-133d91538fbc-kube-api-access-zkgsb" (OuterVolumeSpecName: "kube-api-access-zkgsb") pod "5d16c9bc-9cee-4761-b59d-133d91538fbc" (UID: "5d16c9bc-9cee-4761-b59d-133d91538fbc"). InnerVolumeSpecName "kube-api-access-zkgsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:30:35.258368 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.258346 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-util" (OuterVolumeSpecName: "util") pod "5d16c9bc-9cee-4761-b59d-133d91538fbc" (UID: "5d16c9bc-9cee-4761-b59d-133d91538fbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:30:35.354119 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.354041 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkgsb\" (UniqueName: \"kubernetes.io/projected/5d16c9bc-9cee-4761-b59d-133d91538fbc-kube-api-access-zkgsb\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:35.354119 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.354066 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-util\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:35.354119 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:35.354075 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d16c9bc-9cee-4761-b59d-133d91538fbc-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:36.076761 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.076726 2563 generic.go:356] "Generic (PLEG): container finished" podID="cba30917-00d2-477b-8a8d-9428d179dc34" containerID="fe104162ed1c8e01891914f9e7a908ea01420ef73b70211aff56cf0ae704adb8" exitCode=0 Jul 21 08:30:36.077150 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.076810 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" event={"ID":"cba30917-00d2-477b-8a8d-9428d179dc34","Type":"ContainerDied","Data":"fe104162ed1c8e01891914f9e7a908ea01420ef73b70211aff56cf0ae704adb8"} Jul 21 08:30:36.078530 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.078510 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" Jul 21 08:30:36.078658 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.078531 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn" event={"ID":"5d16c9bc-9cee-4761-b59d-133d91538fbc","Type":"ContainerDied","Data":"30c1e8badd1b8b5d1630c6341270ccf4863da3f818c47a8dbd4be929c444f689"} Jul 21 08:30:36.078658 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.078571 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30c1e8badd1b8b5d1630c6341270ccf4863da3f818c47a8dbd4be929c444f689" Jul 21 08:30:36.214233 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.214211 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:36.250113 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.250090 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:36.263900 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.263876 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-bundle\") pod \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " Jul 21 08:30:36.264040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.263931 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kw27\" (UniqueName: \"kubernetes.io/projected/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-kube-api-access-6kw27\") pod \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " Jul 21 08:30:36.264040 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.263965 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-util\") pod \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\" (UID: \"b85e6820-7dbb-4da0-8fbd-809f3b06bc61\") " Jul 21 08:30:36.264517 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.264494 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-bundle" (OuterVolumeSpecName: "bundle") pod "b85e6820-7dbb-4da0-8fbd-809f3b06bc61" (UID: "b85e6820-7dbb-4da0-8fbd-809f3b06bc61"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:30:36.266361 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.266331 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-kube-api-access-6kw27" (OuterVolumeSpecName: "kube-api-access-6kw27") pod "b85e6820-7dbb-4da0-8fbd-809f3b06bc61" (UID: "b85e6820-7dbb-4da0-8fbd-809f3b06bc61"). InnerVolumeSpecName "kube-api-access-6kw27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:30:36.269846 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.269822 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-util" (OuterVolumeSpecName: "util") pod "b85e6820-7dbb-4da0-8fbd-809f3b06bc61" (UID: "b85e6820-7dbb-4da0-8fbd-809f3b06bc61"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:30:36.364330 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.364301 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-util\") pod \"6f46db13-b76f-412d-bc4b-1d4477609776\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " Jul 21 08:30:36.364473 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.364337 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wzv8\" (UniqueName: \"kubernetes.io/projected/6f46db13-b76f-412d-bc4b-1d4477609776-kube-api-access-5wzv8\") pod \"6f46db13-b76f-412d-bc4b-1d4477609776\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " Jul 21 08:30:36.364473 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.364406 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-bundle\") pod \"6f46db13-b76f-412d-bc4b-1d4477609776\" (UID: \"6f46db13-b76f-412d-bc4b-1d4477609776\") " Jul 21 08:30:36.364661 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.364645 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:36.364704 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.364670 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kw27\" (UniqueName: \"kubernetes.io/projected/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-kube-api-access-6kw27\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:36.364704 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.364685 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b85e6820-7dbb-4da0-8fbd-809f3b06bc61-util\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:36.364827 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.364807 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-bundle" (OuterVolumeSpecName: "bundle") pod "6f46db13-b76f-412d-bc4b-1d4477609776" (UID: "6f46db13-b76f-412d-bc4b-1d4477609776"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:30:36.366723 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.366690 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f46db13-b76f-412d-bc4b-1d4477609776-kube-api-access-5wzv8" (OuterVolumeSpecName: "kube-api-access-5wzv8") pod "6f46db13-b76f-412d-bc4b-1d4477609776" (UID: "6f46db13-b76f-412d-bc4b-1d4477609776"). InnerVolumeSpecName "kube-api-access-5wzv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:30:36.369901 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.369876 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-util" (OuterVolumeSpecName: "util") pod "6f46db13-b76f-412d-bc4b-1d4477609776" (UID: "6f46db13-b76f-412d-bc4b-1d4477609776"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:30:36.465517 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.465443 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-util\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:36.465517 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.465469 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzv8\" (UniqueName: \"kubernetes.io/projected/6f46db13-b76f-412d-bc4b-1d4477609776-kube-api-access-5wzv8\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:36.465517 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:36.465480 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f46db13-b76f-412d-bc4b-1d4477609776-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:37.085709 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.085681 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" Jul 21 08:30:37.085709 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.085699 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp" event={"ID":"6f46db13-b76f-412d-bc4b-1d4477609776","Type":"ContainerDied","Data":"151379b2f6e456063962054c19377c641988ae1f0ec0785ceb70cca14f3827e2"} Jul 21 08:30:37.086188 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.085745 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="151379b2f6e456063962054c19377c641988ae1f0ec0785ceb70cca14f3827e2" Jul 21 08:30:37.087572 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.087552 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" event={"ID":"b85e6820-7dbb-4da0-8fbd-809f3b06bc61","Type":"ContainerDied","Data":"b5532c9af38e7b883dc719a3d1eb0d63c9433a811316ec1adc523f38cc52f8f6"} Jul 21 08:30:37.087678 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.087585 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5532c9af38e7b883dc719a3d1eb0d63c9433a811316ec1adc523f38cc52f8f6" Jul 21 08:30:37.087720 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.087688 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c" Jul 21 08:30:37.217912 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.217890 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:37.273355 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.273329 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp44z\" (UniqueName: \"kubernetes.io/projected/cba30917-00d2-477b-8a8d-9428d179dc34-kube-api-access-wp44z\") pod \"cba30917-00d2-477b-8a8d-9428d179dc34\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " Jul 21 08:30:37.273564 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.273403 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-bundle\") pod \"cba30917-00d2-477b-8a8d-9428d179dc34\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " Jul 21 08:30:37.273564 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.273522 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-util\") pod \"cba30917-00d2-477b-8a8d-9428d179dc34\" (UID: \"cba30917-00d2-477b-8a8d-9428d179dc34\") " Jul 21 08:30:37.273903 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.273876 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-bundle" (OuterVolumeSpecName: "bundle") pod "cba30917-00d2-477b-8a8d-9428d179dc34" (UID: "cba30917-00d2-477b-8a8d-9428d179dc34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:30:37.275587 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.275565 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba30917-00d2-477b-8a8d-9428d179dc34-kube-api-access-wp44z" (OuterVolumeSpecName: "kube-api-access-wp44z") pod "cba30917-00d2-477b-8a8d-9428d179dc34" (UID: "cba30917-00d2-477b-8a8d-9428d179dc34"). InnerVolumeSpecName "kube-api-access-wp44z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:30:37.281189 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.281166 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-util" (OuterVolumeSpecName: "util") pod "cba30917-00d2-477b-8a8d-9428d179dc34" (UID: "cba30917-00d2-477b-8a8d-9428d179dc34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:30:37.374464 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.374429 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp44z\" (UniqueName: \"kubernetes.io/projected/cba30917-00d2-477b-8a8d-9428d179dc34-kube-api-access-wp44z\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:37.374464 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.374458 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:37.374464 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:37.374471 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cba30917-00d2-477b-8a8d-9428d179dc34-util\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:30:38.094893 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:38.094855 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" event={"ID":"cba30917-00d2-477b-8a8d-9428d179dc34","Type":"ContainerDied","Data":"04c0b0f912f7d87f654378f263654fe41e6eee0fff6898402780c0c3711ae733"} Jul 21 08:30:38.095363 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:38.094908 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c0b0f912f7d87f654378f263654fe41e6eee0fff6898402780c0c3711ae733" Jul 21 08:30:38.095363 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:38.094867 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n" Jul 21 08:30:39.735076 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.734993 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d88bb864f-clkqf"] Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735318 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerName="extract" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735329 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerName="extract" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735341 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f46db13-b76f-412d-bc4b-1d4477609776" containerName="extract" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735346 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f46db13-b76f-412d-bc4b-1d4477609776" containerName="extract" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735354 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f46db13-b76f-412d-bc4b-1d4477609776" containerName="pull" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735359 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f46db13-b76f-412d-bc4b-1d4477609776" containerName="pull" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735365 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f46db13-b76f-412d-bc4b-1d4477609776" containerName="util" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735370 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f46db13-b76f-412d-bc4b-1d4477609776" containerName="util" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735376 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba30917-00d2-477b-8a8d-9428d179dc34" containerName="util" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735394 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba30917-00d2-477b-8a8d-9428d179dc34" containerName="util" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735400 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba30917-00d2-477b-8a8d-9428d179dc34" containerName="pull" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735405 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba30917-00d2-477b-8a8d-9428d179dc34" containerName="pull" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735417 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerName="pull" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735423 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerName="pull" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735431 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerName="pull" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735436 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerName="pull" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735444 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerName="util" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735448 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerName="util" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735454 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerName="extract" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735459 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerName="extract" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735465 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerName="util" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735470 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerName="util" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735475 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba30917-00d2-477b-8a8d-9428d179dc34" containerName="extract" Jul 21 08:30:39.735504 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735480 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba30917-00d2-477b-8a8d-9428d179dc34" containerName="extract" Jul 21 08:30:39.736407 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735530 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d16c9bc-9cee-4761-b59d-133d91538fbc" containerName="extract" Jul 21 08:30:39.736407 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735537 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85e6820-7dbb-4da0-8fbd-809f3b06bc61" containerName="extract" Jul 21 08:30:39.736407 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735543 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f46db13-b76f-412d-bc4b-1d4477609776" containerName="extract" Jul 21 08:30:39.736407 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.735551 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba30917-00d2-477b-8a8d-9428d179dc34" containerName="extract" Jul 21 08:30:39.738555 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.738538 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.752544 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.752513 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d88bb864f-clkqf"] Jul 21 08:30:39.794583 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.794543 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-service-ca\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.794583 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.794584 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-console-config\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.794771 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.794646 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-trusted-ca-bundle\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.794771 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.794687 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d5e055d-5c1b-4a00-9530-b4c6713426c3-console-serving-cert\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.794771 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.794711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d5e055d-5c1b-4a00-9530-b4c6713426c3-console-oauth-config\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.794930 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.794773 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-oauth-serving-cert\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.794930 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.794798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktms\" (UniqueName: \"kubernetes.io/projected/0d5e055d-5c1b-4a00-9530-b4c6713426c3-kube-api-access-lktms\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.800296 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.800251 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:39.896202 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.896168 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktms\" (UniqueName: \"kubernetes.io/projected/0d5e055d-5c1b-4a00-9530-b4c6713426c3-kube-api-access-lktms\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.896353 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.896221 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-service-ca\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.896353 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.896337 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-console-config\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.896459 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.896432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-trusted-ca-bundle\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.896500 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.896465 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d5e055d-5c1b-4a00-9530-b4c6713426c3-console-serving-cert\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.896500 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.896494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d5e055d-5c1b-4a00-9530-b4c6713426c3-console-oauth-config\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.896598 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.896548 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-oauth-serving-cert\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.897053 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.897024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-console-config\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.897053 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.897044 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-service-ca\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.897352 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.897165 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-oauth-serving-cert\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.897352 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.897202 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5e055d-5c1b-4a00-9530-b4c6713426c3-trusted-ca-bundle\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.899084 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.899056 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d5e055d-5c1b-4a00-9530-b4c6713426c3-console-serving-cert\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.899199 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.899093 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d5e055d-5c1b-4a00-9530-b4c6713426c3-console-oauth-config\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:39.908813 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:39.908790 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktms\" (UniqueName: \"kubernetes.io/projected/0d5e055d-5c1b-4a00-9530-b4c6713426c3-kube-api-access-lktms\") pod \"console-6d88bb864f-clkqf\" (UID: \"0d5e055d-5c1b-4a00-9530-b4c6713426c3\") " pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:40.048437 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:40.048328 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:40.182413 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:40.182370 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d88bb864f-clkqf"] Jul 21 08:30:40.184142 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:30:40.184116 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5e055d_5c1b_4a00_9530_b4c6713426c3.slice/crio-4c8d10f1d5a0ad2c3de75946d1c202b861f8ca4bb377c5d8bb67b2b01e44de2e WatchSource:0}: Error finding container 4c8d10f1d5a0ad2c3de75946d1c202b861f8ca4bb377c5d8bb67b2b01e44de2e: Status 404 returned error can't find the container with id 4c8d10f1d5a0ad2c3de75946d1c202b861f8ca4bb377c5d8bb67b2b01e44de2e Jul 21 08:30:41.113457 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:41.113417 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d88bb864f-clkqf" event={"ID":"0d5e055d-5c1b-4a00-9530-b4c6713426c3","Type":"ContainerStarted","Data":"ccc1f1fb8ac98e89778f7019dc7eb86dc2c52265e2311cee92fd9ea6237c3802"} Jul 21 08:30:41.113854 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:41.113478 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d88bb864f-clkqf" event={"ID":"0d5e055d-5c1b-4a00-9530-b4c6713426c3","Type":"ContainerStarted","Data":"4c8d10f1d5a0ad2c3de75946d1c202b861f8ca4bb377c5d8bb67b2b01e44de2e"} Jul 21 08:30:41.131962 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:41.131909 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d88bb864f-clkqf" podStartSLOduration=2.131891424 podStartE2EDuration="2.131891424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:30:41.131033449 +0000 UTC m=+713.210071793" watchObservedRunningTime="2026-07-21 08:30:41.131891424 +0000 UTC m=+713.210929755" Jul 21 08:30:44.195407 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:44.195355 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:48.040800 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:48.040767 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:49.800279 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:49.800249 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:50.048465 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:50.048424 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:50.048655 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:50.048509 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:50.053681 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:50.053628 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:50.160923 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:50.160887 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d88bb864f-clkqf" Jul 21 08:30:50.210305 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:50.210274 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7787b59fcd-dvpcc"] Jul 21 08:30:50.334115 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:50.334023 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:50.434313 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:50.434285 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:50.534351 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:50.534323 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:50.635710 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:50.635631 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:54.048631 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:54.048601 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:54.141457 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:54.141427 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:54.799401 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:54.799347 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:55.506914 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:55.506882 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:55.635813 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:55.635787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:55.750931 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:55.750901 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:30:57.030565 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.030529 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4"] Jul 21 08:30:57.034098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.034079 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" Jul 21 08:30:57.036832 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.036810 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pbqsx\"" Jul 21 08:30:57.036832 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.036830 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 08:30:57.037558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.037541 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 08:30:57.037613 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.037572 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 08:30:57.045414 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.045365 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4"] Jul 21 08:30:57.152782 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.152753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgswk\" (UniqueName: \"kubernetes.io/projected/83f55247-7642-48ab-abf8-5c829f324abf-kube-api-access-wgswk\") pod \"dns-operator-controller-manager-864dd78c4c-lndf4\" (UID: \"83f55247-7642-48ab-abf8-5c829f324abf\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" Jul 21 08:30:57.253841 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.253806 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgswk\" (UniqueName: \"kubernetes.io/projected/83f55247-7642-48ab-abf8-5c829f324abf-kube-api-access-wgswk\") pod \"dns-operator-controller-manager-864dd78c4c-lndf4\" (UID: \"83f55247-7642-48ab-abf8-5c829f324abf\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" Jul 21 08:30:57.262395 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.262358 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgswk\" (UniqueName: \"kubernetes.io/projected/83f55247-7642-48ab-abf8-5c829f324abf-kube-api-access-wgswk\") pod \"dns-operator-controller-manager-864dd78c4c-lndf4\" (UID: \"83f55247-7642-48ab-abf8-5c829f324abf\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" Jul 21 08:30:57.344651 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.344549 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" Jul 21 08:30:57.480274 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:57.480246 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4"] Jul 21 08:30:57.483551 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:30:57.483525 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f55247_7642_48ab_abf8_5c829f324abf.slice/crio-5d918d30b3db87e4427606b8d9380202860a71b9be1ba769a77a415ac5ecd3ab WatchSource:0}: Error finding container 5d918d30b3db87e4427606b8d9380202860a71b9be1ba769a77a415ac5ecd3ab: Status 404 returned error can't find the container with id 5d918d30b3db87e4427606b8d9380202860a71b9be1ba769a77a415ac5ecd3ab Jul 21 08:30:58.197362 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:58.197327 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" event={"ID":"83f55247-7642-48ab-abf8-5c829f324abf","Type":"ContainerStarted","Data":"5d918d30b3db87e4427606b8d9380202860a71b9be1ba769a77a415ac5ecd3ab"} Jul 21 08:30:59.223790 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.223759 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6"] Jul 21 08:30:59.226938 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.226918 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.229268 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.229245 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 21 08:30:59.229585 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.229568 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 21 08:30:59.229645 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.229571 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cc662\"" Jul 21 08:30:59.234660 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.234637 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6"] Jul 21 08:30:59.370277 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.370245 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcvrd\" (UniqueName: \"kubernetes.io/projected/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-kube-api-access-mcvrd\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.370497 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.370320 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.370497 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.370443 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.471674 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.471635 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvrd\" (UniqueName: \"kubernetes.io/projected/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-kube-api-access-mcvrd\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.471860 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.471696 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.471860 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.471757 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.471969 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:30:59.471869 2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 21 08:30:59.471969 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:30:59.471938 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-plugin-serving-cert podName:7a90306f-e1ff-48fc-8a80-edd9386fb7ce nodeName:}" failed. No retries permitted until 2026-07-21 08:30:59.971918765 +0000 UTC m=+732.050957074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-mhpf6" (UID: "7a90306f-e1ff-48fc-8a80-edd9386fb7ce") : secret "plugin-serving-cert" not found Jul 21 08:30:59.472471 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.472446 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.481944 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.481876 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvrd\" (UniqueName: \"kubernetes.io/projected/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-kube-api-access-mcvrd\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.980940 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.980899 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:30:59.983477 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:30:59.983446 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a90306f-e1ff-48fc-8a80-edd9386fb7ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mhpf6\" (UID: \"7a90306f-e1ff-48fc-8a80-edd9386fb7ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:31:00.139313 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:00.139280 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" Jul 21 08:31:00.211952 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:00.211920 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" event={"ID":"83f55247-7642-48ab-abf8-5c829f324abf","Type":"ContainerStarted","Data":"5893be0964f0c5c1a30ff9bb354047808834698d3e45e211b746df01c3203f96"} Jul 21 08:31:00.212096 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:00.212012 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" Jul 21 08:31:00.229678 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:00.229629 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" podStartSLOduration=1.106762745 podStartE2EDuration="3.229611816s" totalImagesPullingTime="2.122849071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:31:00.226859384 +0000 UTC m=+732.305897716" watchObservedRunningTime="2026-07-21 08:31:00.229611816 +0000 UTC m=+732.308650146" Jul 21 08:31:00.276512 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:00.276485 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6"] Jul 21 08:31:00.278213 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:31:00.278187 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a90306f_e1ff_48fc_8a80_edd9386fb7ce.slice/crio-0270231fb2ac5d3099d1a30a61fa6197581953f232d9d522057e99ac80944ac8 WatchSource:0}: Error finding container 0270231fb2ac5d3099d1a30a61fa6197581953f232d9d522057e99ac80944ac8: Status 404 returned error can't find the container with id 0270231fb2ac5d3099d1a30a61fa6197581953f232d9d522057e99ac80944ac8 Jul 21 08:31:00.290924 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:00.290881 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:00.355529 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:00.355496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:01.221350 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:01.221316 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" event={"ID":"7a90306f-e1ff-48fc-8a80-edd9386fb7ce","Type":"ContainerStarted","Data":"0270231fb2ac5d3099d1a30a61fa6197581953f232d9d522057e99ac80944ac8"} Jul 21 08:31:05.096582 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:05.096549 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:05.203735 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:05.203675 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:07.004916 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:07.004879 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:07.105029 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:07.104999 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:11.223886 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:11.223621 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lndf4" Jul 21 08:31:11.830430 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:11.830361 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:14.584241 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:14.584198 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:15.237114 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:15.237045 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7787b59fcd-dvpcc" podUID="816776a3-d665-4c28-82d1-e6e38c7eb31e" containerName="console" containerID="cri-o://08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355" gracePeriod=15 Jul 21 08:31:17.261634 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:17.261599 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:17.639571 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:17.639540 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:17.693952 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:17.693917 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:19.005421 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:19.005364 2563 patch_prober.go:28] interesting pod/console-7787b59fcd-dvpcc container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" start-of-body= Jul 21 08:31:19.005914 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:19.005450 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7787b59fcd-dvpcc" podUID="816776a3-d665-4c28-82d1-e6e38c7eb31e" containerName="console" probeResult="failure" output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" Jul 21 08:31:19.809739 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:19.809695 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:19.877825 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:19.877769 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:22.981677 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:22.981655 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:31:23.090961 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.090871 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-oauth-serving-cert\") pod \"816776a3-d665-4c28-82d1-e6e38c7eb31e\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " Jul 21 08:31:23.090961 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.090906 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-config\") pod \"816776a3-d665-4c28-82d1-e6e38c7eb31e\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " Jul 21 08:31:23.090961 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.090958 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l64rf\" (UniqueName: \"kubernetes.io/projected/816776a3-d665-4c28-82d1-e6e38c7eb31e-kube-api-access-l64rf\") pod \"816776a3-d665-4c28-82d1-e6e38c7eb31e\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " Jul 21 08:31:23.091261 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.090988 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-service-ca\") pod \"816776a3-d665-4c28-82d1-e6e38c7eb31e\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " Jul 21 08:31:23.091261 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.091016 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-oauth-config\") pod \"816776a3-d665-4c28-82d1-e6e38c7eb31e\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " Jul 21 08:31:23.091261 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.091046 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-serving-cert\") pod \"816776a3-d665-4c28-82d1-e6e38c7eb31e\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " Jul 21 08:31:23.091261 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.091088 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-trusted-ca-bundle\") pod \"816776a3-d665-4c28-82d1-e6e38c7eb31e\" (UID: \"816776a3-d665-4c28-82d1-e6e38c7eb31e\") " Jul 21 08:31:23.091491 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.091259 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "816776a3-d665-4c28-82d1-e6e38c7eb31e" (UID: "816776a3-d665-4c28-82d1-e6e38c7eb31e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:31:23.091491 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.091348 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-oauth-serving-cert\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:31:23.091491 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.091374 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-service-ca" (OuterVolumeSpecName: "service-ca") pod "816776a3-d665-4c28-82d1-e6e38c7eb31e" (UID: "816776a3-d665-4c28-82d1-e6e38c7eb31e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:31:23.091491 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.091430 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-config" (OuterVolumeSpecName: "console-config") pod "816776a3-d665-4c28-82d1-e6e38c7eb31e" (UID: "816776a3-d665-4c28-82d1-e6e38c7eb31e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:31:23.091672 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.091652 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "816776a3-d665-4c28-82d1-e6e38c7eb31e" (UID: "816776a3-d665-4c28-82d1-e6e38c7eb31e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:31:23.093310 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.093277 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "816776a3-d665-4c28-82d1-e6e38c7eb31e" (UID: "816776a3-d665-4c28-82d1-e6e38c7eb31e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:31:23.093585 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.093558 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "816776a3-d665-4c28-82d1-e6e38c7eb31e" (UID: "816776a3-d665-4c28-82d1-e6e38c7eb31e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:31:23.093639 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.093589 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816776a3-d665-4c28-82d1-e6e38c7eb31e-kube-api-access-l64rf" (OuterVolumeSpecName: "kube-api-access-l64rf") pod "816776a3-d665-4c28-82d1-e6e38c7eb31e" (UID: "816776a3-d665-4c28-82d1-e6e38c7eb31e"). InnerVolumeSpecName "kube-api-access-l64rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:31:23.192661 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.192627 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l64rf\" (UniqueName: \"kubernetes.io/projected/816776a3-d665-4c28-82d1-e6e38c7eb31e-kube-api-access-l64rf\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:31:23.192661 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.192654 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-service-ca\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:31:23.192661 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.192664 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-oauth-config\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:31:23.192898 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.192674 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-serving-cert\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:31:23.192898 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.192682 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-trusted-ca-bundle\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:31:23.192898 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.192690 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/816776a3-d665-4c28-82d1-e6e38c7eb31e-console-config\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:31:23.348037 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.347924 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" event={"ID":"7a90306f-e1ff-48fc-8a80-edd9386fb7ce","Type":"ContainerStarted","Data":"3b8d30be0ede0e49dea4b06f29e84577f5f347457918e794f87993885d74d6f5"} Jul 21 08:31:23.349426 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.349401 2563 generic.go:356] "Generic (PLEG): container finished" podID="816776a3-d665-4c28-82d1-e6e38c7eb31e" containerID="08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355" exitCode=0 Jul 21 08:31:23.349426 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.349423 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7787b59fcd-dvpcc" event={"ID":"816776a3-d665-4c28-82d1-e6e38c7eb31e","Type":"ContainerDied","Data":"08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355"} Jul 21 08:31:23.349629 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.349467 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7787b59fcd-dvpcc" Jul 21 08:31:23.349629 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.349484 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7787b59fcd-dvpcc" event={"ID":"816776a3-d665-4c28-82d1-e6e38c7eb31e","Type":"ContainerDied","Data":"c25eacad994cb833d1e18540819080f2271b04b71ef34b8272ce3e17b28d84ad"} Jul 21 08:31:23.349629 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.349504 2563 scope.go:117] "RemoveContainer" containerID="08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355" Jul 21 08:31:23.358373 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.358350 2563 scope.go:117] "RemoveContainer" containerID="08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355" Jul 21 08:31:23.358821 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:31:23.358792 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355\": container with ID starting with 08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355 not found: ID does not exist" containerID="08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355" Jul 21 08:31:23.358906 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.358830 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355"} err="failed to get container status \"08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355\": rpc error: code = NotFound desc = could not find container \"08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355\": container with ID starting with 08585a72a1e51a9ac4cc8f92ad4db1386bf1e7f64cadca301025352d6efc1355 not found: ID does not exist" Jul 21 08:31:23.365872 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.365834 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mhpf6" podStartSLOduration=1.7557030729999998 podStartE2EDuration="24.365821235s" totalImagesPullingTime="22.610118162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:31:23.365049439 +0000 UTC m=+755.444087762" watchObservedRunningTime="2026-07-21 08:31:23.365821235 +0000 UTC m=+755.444859566" Jul 21 08:31:23.385993 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.385966 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7787b59fcd-dvpcc"] Jul 21 08:31:23.393333 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:23.393310 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7787b59fcd-dvpcc"] Jul 21 08:31:24.357211 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:24.357176 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="816776a3-d665-4c28-82d1-e6e38c7eb31e" path="/var/lib/kubelet/pods/816776a3-d665-4c28-82d1-e6e38c7eb31e/volumes" Jul 21 08:31:26.635602 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:26.635564 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:26.736474 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:26.736438 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:31.437321 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:31.437282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:31.537736 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:31.537698 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:33.895434 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:33.895363 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:33.955781 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:33.955755 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:39.639530 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:39.639493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:39.699638 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:39.699607 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:43.390277 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:43.390244 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:43.558132 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:43.558093 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:45.780838 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:45.780802 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:45.918699 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:45.918668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:46.019596 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.019550 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:46.209129 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.209101 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:31:46.209541 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.209522 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816776a3-d665-4c28-82d1-e6e38c7eb31e" containerName="console" Jul 21 08:31:46.209541 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.209541 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="816776a3-d665-4c28-82d1-e6e38c7eb31e" containerName="console" Jul 21 08:31:46.209715 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.209627 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="816776a3-d665-4c28-82d1-e6e38c7eb31e" containerName="console" Jul 21 08:31:46.212207 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.212189 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:31:46.214393 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.214364 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 08:31:46.220674 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.220651 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:31:46.249776 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.249755 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:31:46.400099 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.400069 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3e681ce3-b040-42ba-825b-18ed78d49fad-config-file\") pod \"limitador-limitador-54d6d988d5-rm5h8\" (UID: \"3e681ce3-b040-42ba-825b-18ed78d49fad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:31:46.400279 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.400138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvvl\" (UniqueName: \"kubernetes.io/projected/3e681ce3-b040-42ba-825b-18ed78d49fad-kube-api-access-stvvl\") pod \"limitador-limitador-54d6d988d5-rm5h8\" (UID: \"3e681ce3-b040-42ba-825b-18ed78d49fad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:31:46.501273 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.501196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvvl\" (UniqueName: \"kubernetes.io/projected/3e681ce3-b040-42ba-825b-18ed78d49fad-kube-api-access-stvvl\") pod \"limitador-limitador-54d6d988d5-rm5h8\" (UID: \"3e681ce3-b040-42ba-825b-18ed78d49fad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:31:46.501273 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.501251 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3e681ce3-b040-42ba-825b-18ed78d49fad-config-file\") pod \"limitador-limitador-54d6d988d5-rm5h8\" (UID: \"3e681ce3-b040-42ba-825b-18ed78d49fad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:31:46.501904 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.501886 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3e681ce3-b040-42ba-825b-18ed78d49fad-config-file\") pod \"limitador-limitador-54d6d988d5-rm5h8\" (UID: \"3e681ce3-b040-42ba-825b-18ed78d49fad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:31:46.508960 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.508936 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvvl\" (UniqueName: \"kubernetes.io/projected/3e681ce3-b040-42ba-825b-18ed78d49fad-kube-api-access-stvvl\") pod \"limitador-limitador-54d6d988d5-rm5h8\" (UID: \"3e681ce3-b040-42ba-825b-18ed78d49fad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:31:46.522553 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.522525 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:31:46.615806 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.615774 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:46.662529 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.662505 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:31:46.664333 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:31:46.664311 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e681ce3_b040_42ba_825b_18ed78d49fad.slice/crio-fff34aba1b82dd8d25db6d11fd59b18543c420d1cf9dc8024fb45c6e9e48b23a WatchSource:0}: Error finding container fff34aba1b82dd8d25db6d11fd59b18543c420d1cf9dc8024fb45c6e9e48b23a: Status 404 returned error can't find the container with id fff34aba1b82dd8d25db6d11fd59b18543c420d1cf9dc8024fb45c6e9e48b23a Jul 21 08:31:46.677908 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:46.677887 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:47.468859 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:47.468820 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" event={"ID":"3e681ce3-b040-42ba-825b-18ed78d49fad","Type":"ContainerStarted","Data":"fff34aba1b82dd8d25db6d11fd59b18543c420d1cf9dc8024fb45c6e9e48b23a"} Jul 21 08:31:47.665124 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:47.665095 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:47.767520 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:47.767420 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:49.482599 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:49.482562 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" event={"ID":"3e681ce3-b040-42ba-825b-18ed78d49fad","Type":"ContainerStarted","Data":"2ffe330e2189ee23223efdd900bda126b18fbac70d161e3668a8c203ba8a4465"} Jul 21 08:31:49.483025 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:49.482624 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:31:49.499554 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:49.499504 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" podStartSLOduration=1.062101345 podStartE2EDuration="3.499490472s" totalImagesPullingTime="2.437389127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:31:49.499218852 +0000 UTC m=+781.578257186" watchObservedRunningTime="2026-07-21 08:31:49.499490472 +0000 UTC m=+781.578528803" Jul 21 08:31:52.235741 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:52.235700 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:52.295950 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:52.295913 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:52.648956 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:52.648916 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:52.749657 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:52.749623 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:56.439232 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:56.439201 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:31:59.136349 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:31:59.136315 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:00.488881 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:00.488853 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-rm5h8" Jul 21 08:32:01.971200 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:01.971168 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:02.076606 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:02.076549 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:02.477771 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:02.477737 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:15.099488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:15.099371 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:15.422265 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:15.422235 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:24.611214 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:24.611168 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:24.720912 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:24.720873 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:24.780637 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:24.780597 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:24.930175 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:24.930143 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:28.286582 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:28.286552 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:28.386335 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:28.386307 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:31.387618 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:31.387584 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:31.453367 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:31.453338 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:42.548489 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:42.548453 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:32:42.649680 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:32:42.649646 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:00.273397 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:00.273340 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:00.372481 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:00.372451 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:00.862423 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:00.862359 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:01.041012 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:01.040979 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:01.305319 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:01.305288 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:01.406525 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:01.406489 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:01.483376 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:01.483339 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:33:11.719702 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:11.719659 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:11.818274 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:11.818238 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:17.269063 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:17.269029 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:17.625822 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:17.625742 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:33:26.370124 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:26.370097 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:27.118665 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:27.118631 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:33:29.335444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:29.335375 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:32.552492 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:32.552461 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:33.723077 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:33.723045 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:33:39.143296 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.143208 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7"] Jul 21 08:33:39.146834 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.146814 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.150166 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.150145 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 08:33:39.150266 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.150145 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 08:33:39.150266 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.150147 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k878q\"" Jul 21 08:33:39.150266 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.150147 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 21 08:33:39.154729 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.154703 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7"] Jul 21 08:33:39.158574 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.158546 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.158684 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.158603 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.158684 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.158662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.158811 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.158692 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.158811 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.158719 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.158811 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.158774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nr6d\" (UniqueName: \"kubernetes.io/projected/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-kube-api-access-7nr6d\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.241451 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.241428 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:39.259754 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.259727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.259851 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.259767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.259851 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.259800 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.259851 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.259827 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.259980 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.259853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.259980 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.259883 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nr6d\" (UniqueName: \"kubernetes.io/projected/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-kube-api-access-7nr6d\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.260162 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.260142 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.260230 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.260185 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.260283 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.260245 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.262218 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.262193 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.262648 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.262628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.267435 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.267406 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nr6d\" (UniqueName: \"kubernetes.io/projected/ca6a1159-57d8-46cc-b1d1-61cdb48b23e8-kube-api-access-7nr6d\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7\" (UID: \"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.477506 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.477412 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:39.611779 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:39.611754 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7"] Jul 21 08:33:39.613838 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:33:39.613814 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6a1159_57d8_46cc_b1d1_61cdb48b23e8.slice/crio-bbc03ec5568f867c1991485a3eef73883e2f96853d024dc3f9819a2ef6f3669d WatchSource:0}: Error finding container bbc03ec5568f867c1991485a3eef73883e2f96853d024dc3f9819a2ef6f3669d: Status 404 returned error can't find the container with id bbc03ec5568f867c1991485a3eef73883e2f96853d024dc3f9819a2ef6f3669d Jul 21 08:33:40.036700 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:40.036656 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" event={"ID":"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8","Type":"ContainerStarted","Data":"bbc03ec5568f867c1991485a3eef73883e2f96853d024dc3f9819a2ef6f3669d"} Jul 21 08:33:40.826811 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:40.826779 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:33:41.043866 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:41.043774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" event={"ID":"ca6a1159-57d8-46cc-b1d1-61cdb48b23e8","Type":"ContainerStarted","Data":"e2bbcc3b2cc19c8cc45dd6e23743cd439759b678569d47a86f2e959e58abd9de"} Jul 21 08:33:41.061748 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:41.061701 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" podStartSLOduration=1.037563833 podStartE2EDuration="2.061682668s" totalImagesPullingTime="1.024118835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:33:41.060912039 +0000 UTC m=+893.139950371" watchObservedRunningTime="2026-07-21 08:33:41.061682668 +0000 UTC m=+893.140721000" Jul 21 08:33:45.379790 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:45.379751 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:48.223029 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:48.222991 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:33:49.478072 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:49.478031 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:49.480501 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:49.478114 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:49.491239 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:49.491210 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:50.102330 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:50.102302 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7" Jul 21 08:33:55.525298 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.525265 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb"] Jul 21 08:33:55.531843 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.531822 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.534203 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.534183 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 21 08:33:55.540244 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.540223 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb"] Jul 21 08:33:55.555187 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.555165 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:33:55.600644 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.600610 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.600763 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.600662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.600763 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.600712 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.600763 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.600728 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qlk\" (UniqueName: \"kubernetes.io/projected/3bd2bef4-3248-43ea-bf12-70ed6ad88965-kube-api-access-q9qlk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.600868 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.600838 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.600902 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.600871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd2bef4-3248-43ea-bf12-70ed6ad88965-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.702054 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.702025 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd2bef4-3248-43ea-bf12-70ed6ad88965-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.702229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.702065 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.702229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.702100 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.702229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.702132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.702229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.702149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qlk\" (UniqueName: \"kubernetes.io/projected/3bd2bef4-3248-43ea-bf12-70ed6ad88965-kube-api-access-q9qlk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.702229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.702217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.702606 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.702572 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.702744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.702645 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.702744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.702666 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.704609 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.704588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd2bef4-3248-43ea-bf12-70ed6ad88965-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.704834 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.704817 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd2bef4-3248-43ea-bf12-70ed6ad88965-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.709733 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.709704 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qlk\" (UniqueName: \"kubernetes.io/projected/3bd2bef4-3248-43ea-bf12-70ed6ad88965-kube-api-access-q9qlk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb\" (UID: \"3bd2bef4-3248-43ea-bf12-70ed6ad88965\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.841839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.841745 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:33:55.979789 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.979758 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb"] Jul 21 08:33:55.982919 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:33:55.982896 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bd2bef4_3248_43ea_bf12_70ed6ad88965.slice/crio-68f13927b7fdd1d42efaef0317447da13db5ee351c7d1857c3c233148f001867 WatchSource:0}: Error finding container 68f13927b7fdd1d42efaef0317447da13db5ee351c7d1857c3c233148f001867: Status 404 returned error can't find the container with id 68f13927b7fdd1d42efaef0317447da13db5ee351c7d1857c3c233148f001867 Jul 21 08:33:55.984309 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:55.984287 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:33:56.128496 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:56.128407 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" event={"ID":"3bd2bef4-3248-43ea-bf12-70ed6ad88965","Type":"ContainerStarted","Data":"68f13927b7fdd1d42efaef0317447da13db5ee351c7d1857c3c233148f001867"} Jul 21 08:33:57.135328 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:57.135290 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" event={"ID":"3bd2bef4-3248-43ea-bf12-70ed6ad88965","Type":"ContainerStarted","Data":"8ed49c4914029202f646dd2ac355ae7fbdd77b971640cfd731caeb1b551023f4"} Jul 21 08:33:57.151500 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:57.151455 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" podStartSLOduration=1.87174539 podStartE2EDuration="2.151438253s" totalImagesPullingTime="279.692863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:33:57.151160011 +0000 UTC m=+909.230198342" watchObservedRunningTime="2026-07-21 08:33:57.151438253 +0000 UTC m=+909.230476583" Jul 21 08:33:58.226108 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:33:58.226073 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:34:05.842797 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:34:05.842756 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:34:05.843437 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:34:05.842812 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:34:05.855376 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:34:05.855352 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:34:06.189442 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:34:06.189414 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb" Jul 21 08:34:56.234991 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:34:56.234954 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:34:56.333561 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:34:56.333521 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:34:57.440326 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:34:57.440294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:34:57.545404 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:34:57.545361 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:35:04.546431 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:35:04.546375 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:35:35.959877 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:35:35.959841 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:35:47.550939 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:35:47.550904 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:35:57.032681 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:35:57.032644 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:36:50.983283 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:36:50.983254 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:36:51.068059 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:36:51.068022 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:37:11.008584 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:11.008541 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:37:11.078662 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:11.078627 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:37:12.754336 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:12.754272 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:37:14.234950 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:14.234922 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:37:35.543509 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:35.543472 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:37:47.543098 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:47.543064 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:37:52.913604 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:52.913569 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:37:53.002673 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:53.002641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:37:53.142027 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:53.141998 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:37:59.960992 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:37:59.960963 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:38:00.239458 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:38:00.239344 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:38:00.421437 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:38:00.421369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:38:00.692553 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:38:00.692524 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:38:07.634711 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:38:07.634672 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:38:07.857451 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:38:07.857424 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:38:24.445734 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:38:24.445700 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:38:34.348728 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:38:34.348693 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:38:43.546738 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:38:43.546709 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:38:55.141680 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:38:55.141648 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:39:06.143989 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:39:06.143953 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:39:15.426800 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:39:15.426761 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:39:26.633242 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:39:26.633206 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:39:36.323178 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:39:36.323141 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:39:59.224140 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:39:59.224104 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:40:00.543567 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:40:00.543535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:40:03.328525 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:40:03.328496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:40:04.523316 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:40:04.523272 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:40:08.222155 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:40:08.222113 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:40:32.942301 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:40:32.942264 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:40:44.743592 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:40:44.743559 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:41:00.140175 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:41:00.140141 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:41:01.247177 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:41:01.247139 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:41:01.347958 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:41:01.347927 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:41:09.438751 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:41:09.438664 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:41:20.730949 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:41:20.730911 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:41:30.155825 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:41:30.155788 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:41:39.829551 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:41:39.829514 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:43:29.293480 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:43:29.293447 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:43:38.138474 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:43:38.138438 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:43:50.434143 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:43:50.434104 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:44:02.542012 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:44:02.541976 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:44:12.435997 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:44:12.435903 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:44:23.237644 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:44:23.237608 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:44:26.792969 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:44:26.792936 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:44:33.540905 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:44:33.540870 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:44:43.661045 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:44:43.661008 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:44:56.138580 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:44:56.138541 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:45:05.033177 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:45:05.033137 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:45:15.628603 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:45:15.628566 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:45:37.584292 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:45:37.584255 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:45:46.853315 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:45:46.853233 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:46:00.863549 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:00.863511 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr"] Jul 21 08:46:00.866736 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:00.866720 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:00.871240 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:00.871224 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ee93347b-kserve-self-signed-certs\"" Jul 21 08:46:00.877982 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:00.877962 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr"] Jul 21 08:46:00.928026 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:00.927999 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:46:01.008839 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.008807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tls-certs\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.009001 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.008862 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-model-cache\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.009001 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.008927 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99knj\" (UniqueName: \"kubernetes.io/projected/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-kube-api-access-99knj\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.009001 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.008975 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-dshm\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.009124 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.009002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tmp-dir\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.009124 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.009027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-home\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.110170 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.110138 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-model-cache\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.110329 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.110186 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99knj\" (UniqueName: \"kubernetes.io/projected/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-kube-api-access-99knj\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.110329 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.110224 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-dshm\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.110329 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.110252 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tmp-dir\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.110329 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.110287 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-home\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.110329 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.110322 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tls-certs\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.110629 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.110610 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-model-cache\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.110689 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.110663 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tmp-dir\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.110808 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.110786 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-home\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.112717 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.112695 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-dshm\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.112971 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.112953 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tls-certs\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.118840 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.118792 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99knj\" (UniqueName: \"kubernetes.io/projected/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-kube-api-access-99knj\") pod \"e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.175706 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.175676 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:01.311205 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.311177 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr"] Jul 21 08:46:01.313475 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:46:01.313446 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fbb5a7f_9cba_43b0_a516_7c48b139be6b.slice/crio-b2f29b25b936dc5061bb01e15558d1eab6c7fd21f3b86cbb40959e823ec58ed7 WatchSource:0}: Error finding container b2f29b25b936dc5061bb01e15558d1eab6c7fd21f3b86cbb40959e823ec58ed7: Status 404 returned error can't find the container with id b2f29b25b936dc5061bb01e15558d1eab6c7fd21f3b86cbb40959e823ec58ed7 Jul 21 08:46:01.771686 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.771650 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" event={"ID":"6fbb5a7f-9cba-43b0-a516-7c48b139be6b","Type":"ContainerStarted","Data":"8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5"} Jul 21 08:46:01.771686 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.771688 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" event={"ID":"6fbb5a7f-9cba-43b0-a516-7c48b139be6b","Type":"ContainerStarted","Data":"b2f29b25b936dc5061bb01e15558d1eab6c7fd21f3b86cbb40959e823ec58ed7"} Jul 21 08:46:01.793092 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:01.793045 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" podStartSLOduration=1.79302867 podStartE2EDuration="1.79302867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:46:01.791144109 +0000 UTC m=+1633.870182441" watchObservedRunningTime="2026-07-21 08:46:01.79302867 +0000 UTC m=+1633.872067061" Jul 21 08:46:04.142223 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:04.142193 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:46:05.870738 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:05.870707 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q"] Jul 21 08:46:05.874178 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:05.874161 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:05.879104 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:05.879083 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ee93347b-kserve-self-signed-certs\"" Jul 21 08:46:05.932964 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:05.932935 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:46:05.954218 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:05.954183 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q"] Jul 21 08:46:06.051206 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.051165 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-model-cache\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.051375 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.051226 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-home\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.051375 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.051300 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqg7\" (UniqueName: \"kubernetes.io/projected/a5c64fea-1604-4ee1-badc-eae9cdcf267b-kube-api-access-kkqg7\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.051375 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.051322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-dshm\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.051375 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.051337 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tmp-dir\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.051375 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.051353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tls-certs\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.152585 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.152549 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-model-cache\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.152726 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.152617 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-home\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.152726 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.152673 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqg7\" (UniqueName: \"kubernetes.io/projected/a5c64fea-1604-4ee1-badc-eae9cdcf267b-kube-api-access-kkqg7\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.152726 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.152703 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-dshm\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.152726 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.152719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tmp-dir\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.152871 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.152740 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tls-certs\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.153036 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.153013 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-model-cache\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.153075 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.153039 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-home\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.153181 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.153153 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tmp-dir\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.155278 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.155249 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-dshm\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.155366 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.155334 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tls-certs\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.162253 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.162234 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqg7\" (UniqueName: \"kubernetes.io/projected/a5c64fea-1604-4ee1-badc-eae9cdcf267b-kube-api-access-kkqg7\") pod \"e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.183674 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.183650 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:06.319410 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.319306 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q"] Jul 21 08:46:06.322680 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:46:06.322647 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5c64fea_1604_4ee1_badc_eae9cdcf267b.slice/crio-e5caf28a1a56467923ffbc6f1a08c2986c6c58970c98f94aef5575d2d8a420c3 WatchSource:0}: Error finding container e5caf28a1a56467923ffbc6f1a08c2986c6c58970c98f94aef5575d2d8a420c3: Status 404 returned error can't find the container with id e5caf28a1a56467923ffbc6f1a08c2986c6c58970c98f94aef5575d2d8a420c3 Jul 21 08:46:06.798478 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.798434 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" event={"ID":"a5c64fea-1604-4ee1-badc-eae9cdcf267b","Type":"ContainerStarted","Data":"b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d"} Jul 21 08:46:06.798478 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.798482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" event={"ID":"a5c64fea-1604-4ee1-badc-eae9cdcf267b","Type":"ContainerStarted","Data":"e5caf28a1a56467923ffbc6f1a08c2986c6c58970c98f94aef5575d2d8a420c3"} Jul 21 08:46:06.837298 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:06.837250 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" podStartSLOduration=1.83723396 podStartE2EDuration="1.83723396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:46:06.83688249 +0000 UTC m=+1638.915920822" watchObservedRunningTime="2026-07-21 08:46:06.83723396 +0000 UTC m=+1638.916272290" Jul 21 08:46:11.176471 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:11.176439 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:11.176893 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:11.176521 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:11.189367 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:11.189341 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:11.841907 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:11.841878 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:16.146229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:16.146199 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:46:16.184638 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:16.184612 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:16.184805 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:16.184673 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:16.197487 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:16.197462 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:16.866597 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:16.866570 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:27.160120 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:27.160087 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr"] Jul 21 08:46:27.188864 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:27.188839 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:46:27.939313 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:27.939278 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q"] Jul 21 08:46:27.956082 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:27.956044 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:46:31.150076 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:31.150039 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:46:42.180746 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.180706 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" podUID="6fbb5a7f-9cba-43b0-a516-7c48b139be6b" containerName="main" containerID="cri-o://8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5" gracePeriod=45 Jul 21 08:46:42.430803 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.430746 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:42.540506 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540475 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-home\") pod \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " Jul 21 08:46:42.540679 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540548 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tmp-dir\") pod \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " Jul 21 08:46:42.540679 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540577 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tls-certs\") pod \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " Jul 21 08:46:42.540679 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540600 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-model-cache\") pod \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " Jul 21 08:46:42.540679 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540643 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99knj\" (UniqueName: \"kubernetes.io/projected/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-kube-api-access-99knj\") pod \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " Jul 21 08:46:42.540905 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540694 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-dshm\") pod \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\" (UID: \"6fbb5a7f-9cba-43b0-a516-7c48b139be6b\") " Jul 21 08:46:42.540905 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540776 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-home" (OuterVolumeSpecName: "home") pod "6fbb5a7f-9cba-43b0-a516-7c48b139be6b" (UID: "6fbb5a7f-9cba-43b0-a516-7c48b139be6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:46:42.540905 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540855 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fbb5a7f-9cba-43b0-a516-7c48b139be6b" (UID: "6fbb5a7f-9cba-43b0-a516-7c48b139be6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:46:42.540905 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540867 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fbb5a7f-9cba-43b0-a516-7c48b139be6b" (UID: "6fbb5a7f-9cba-43b0-a516-7c48b139be6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:46:42.541078 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540962 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:42.541078 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.540993 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:42.541078 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.541002 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:42.543017 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.542982 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-dshm" (OuterVolumeSpecName: "dshm") pod "6fbb5a7f-9cba-43b0-a516-7c48b139be6b" (UID: "6fbb5a7f-9cba-43b0-a516-7c48b139be6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:46:42.543132 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.543066 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fbb5a7f-9cba-43b0-a516-7c48b139be6b" (UID: "6fbb5a7f-9cba-43b0-a516-7c48b139be6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:46:42.543132 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.543080 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-kube-api-access-99knj" (OuterVolumeSpecName: "kube-api-access-99knj") pod "6fbb5a7f-9cba-43b0-a516-7c48b139be6b" (UID: "6fbb5a7f-9cba-43b0-a516-7c48b139be6b"). InnerVolumeSpecName "kube-api-access-99knj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:46:42.641640 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.641604 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:42.641640 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.641634 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99knj\" (UniqueName: \"kubernetes.io/projected/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-kube-api-access-99knj\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:42.641640 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.641643 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fbb5a7f-9cba-43b0-a516-7c48b139be6b-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:42.959707 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.959647 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" podUID="a5c64fea-1604-4ee1-badc-eae9cdcf267b" containerName="main" containerID="cri-o://b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d" gracePeriod=45 Jul 21 08:46:42.992581 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.992549 2563 generic.go:356] "Generic (PLEG): container finished" podID="6fbb5a7f-9cba-43b0-a516-7c48b139be6b" containerID="8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5" exitCode=0 Jul 21 08:46:42.992723 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.992610 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" Jul 21 08:46:42.992723 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.992634 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" event={"ID":"6fbb5a7f-9cba-43b0-a516-7c48b139be6b","Type":"ContainerDied","Data":"8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5"} Jul 21 08:46:42.992723 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.992684 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr" event={"ID":"6fbb5a7f-9cba-43b0-a516-7c48b139be6b","Type":"ContainerDied","Data":"b2f29b25b936dc5061bb01e15558d1eab6c7fd21f3b86cbb40959e823ec58ed7"} Jul 21 08:46:42.992723 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:42.992706 2563 scope.go:117] "RemoveContainer" containerID="8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5" Jul 21 08:46:43.051027 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.051004 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr"] Jul 21 08:46:43.052978 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.052959 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ee93347b-kserve-584cddd85f-76ccr"] Jul 21 08:46:43.070651 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.070634 2563 scope.go:117] "RemoveContainer" containerID="8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5" Jul 21 08:46:43.070945 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:46:43.070927 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5\": container with ID starting with 8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5 not found: ID does not exist" containerID="8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5" Jul 21 08:46:43.071015 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.070953 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5"} err="failed to get container status \"8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5\": rpc error: code = NotFound desc = could not find container \"8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5\": container with ID starting with 8e5ecffdccb6e78451aef48e2385e9532d320f7ed7253948cfd8c7a36a0fe1b5 not found: ID does not exist" Jul 21 08:46:43.210109 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.210049 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:43.246319 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246291 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tmp-dir\") pod \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " Jul 21 08:46:43.246461 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246330 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-model-cache\") pod \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " Jul 21 08:46:43.246461 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246363 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-dshm\") pod \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " Jul 21 08:46:43.246461 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246435 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkqg7\" (UniqueName: \"kubernetes.io/projected/a5c64fea-1604-4ee1-badc-eae9cdcf267b-kube-api-access-kkqg7\") pod \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " Jul 21 08:46:43.246645 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246476 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tls-certs\") pod \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " Jul 21 08:46:43.246645 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246516 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-home\") pod \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\" (UID: \"a5c64fea-1604-4ee1-badc-eae9cdcf267b\") " Jul 21 08:46:43.246645 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246547 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5c64fea-1604-4ee1-badc-eae9cdcf267b" (UID: "a5c64fea-1604-4ee1-badc-eae9cdcf267b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:46:43.246645 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246560 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5c64fea-1604-4ee1-badc-eae9cdcf267b" (UID: "a5c64fea-1604-4ee1-badc-eae9cdcf267b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:46:43.246846 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246744 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:43.246846 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246764 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:43.246846 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.246798 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-home" (OuterVolumeSpecName: "home") pod "a5c64fea-1604-4ee1-badc-eae9cdcf267b" (UID: "a5c64fea-1604-4ee1-badc-eae9cdcf267b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:46:43.248731 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.248708 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5c64fea-1604-4ee1-badc-eae9cdcf267b" (UID: "a5c64fea-1604-4ee1-badc-eae9cdcf267b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:46:43.248848 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.248809 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c64fea-1604-4ee1-badc-eae9cdcf267b-kube-api-access-kkqg7" (OuterVolumeSpecName: "kube-api-access-kkqg7") pod "a5c64fea-1604-4ee1-badc-eae9cdcf267b" (UID: "a5c64fea-1604-4ee1-badc-eae9cdcf267b"). InnerVolumeSpecName "kube-api-access-kkqg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:46:43.249008 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.248992 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-dshm" (OuterVolumeSpecName: "dshm") pod "a5c64fea-1604-4ee1-badc-eae9cdcf267b" (UID: "a5c64fea-1604-4ee1-badc-eae9cdcf267b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:46:43.347369 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.347334 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c64fea-1604-4ee1-badc-eae9cdcf267b-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:43.347369 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.347363 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:43.347369 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.347372 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c64fea-1604-4ee1-badc-eae9cdcf267b-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:43.347622 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:43.347404 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkqg7\" (UniqueName: \"kubernetes.io/projected/a5c64fea-1604-4ee1-badc-eae9cdcf267b-kube-api-access-kkqg7\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:46:44.001123 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.001090 2563 generic.go:356] "Generic (PLEG): container finished" podID="a5c64fea-1604-4ee1-badc-eae9cdcf267b" containerID="b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d" exitCode=0 Jul 21 08:46:44.001291 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.001133 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" Jul 21 08:46:44.001291 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.001177 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" event={"ID":"a5c64fea-1604-4ee1-badc-eae9cdcf267b","Type":"ContainerDied","Data":"b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d"} Jul 21 08:46:44.001291 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.001219 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q" event={"ID":"a5c64fea-1604-4ee1-badc-eae9cdcf267b","Type":"ContainerDied","Data":"e5caf28a1a56467923ffbc6f1a08c2986c6c58970c98f94aef5575d2d8a420c3"} Jul 21 08:46:44.001291 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.001243 2563 scope.go:117] "RemoveContainer" containerID="b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d" Jul 21 08:46:44.010781 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.010765 2563 scope.go:117] "RemoveContainer" containerID="b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d" Jul 21 08:46:44.011034 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:46:44.011016 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d\": container with ID starting with b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d not found: ID does not exist" containerID="b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d" Jul 21 08:46:44.011097 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.011041 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d"} err="failed to get container status \"b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d\": rpc error: code = NotFound desc = could not find container \"b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d\": container with ID starting with b2609aab3e54471ac0a1c1c51377562babe08f3e63b84e3d6fea82bd4a7ff24d not found: ID does not exist" Jul 21 08:46:44.022411 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.022367 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q"] Jul 21 08:46:44.026298 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.026276 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ee93347b-kserve-59c6765f9-mk59q"] Jul 21 08:46:44.358452 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.358354 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fbb5a7f-9cba-43b0-a516-7c48b139be6b" path="/var/lib/kubelet/pods/6fbb5a7f-9cba-43b0-a516-7c48b139be6b/volumes" Jul 21 08:46:44.358811 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.358739 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c64fea-1604-4ee1-badc-eae9cdcf267b" path="/var/lib/kubelet/pods/a5c64fea-1604-4ee1-badc-eae9cdcf267b/volumes" Jul 21 08:46:44.643693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:44.643662 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:46:50.435857 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:50.435820 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:46:50.699421 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:50.699311 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:46:50.758431 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:46:50.758376 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:47:08.454231 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:47:08.454198 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:47:22.926960 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:47:22.926921 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:47:32.246550 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:47:32.246516 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:47:45.748825 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:47:45.748789 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:47:55.733291 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:47:55.733247 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:48:09.465115 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:48:09.465077 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:48:18.090974 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:48:18.090937 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:48:27.950086 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:48:27.950052 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:48:40.131611 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:48:40.131528 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:48:48.927481 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:48:48.927445 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:48:55.434129 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:48:55.434085 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:49:11.850519 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:11.850483 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:49:20.695259 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:20.695225 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:20.993273 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:20.993144 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:23.234747 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:23.234714 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:49:35.954799 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:35.954764 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:49:41.481193 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:41.481154 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:41.583995 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:41.583961 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:41.934931 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:41.934890 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:47.671667 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:47.671633 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:47.815910 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:47.815878 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:47.913236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:47.913182 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:48.276400 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:48.276359 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:48.395606 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:48.395572 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:50.491488 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:50.491451 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:50.572776 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:50.572746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:50.644775 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:50.644743 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:50.744728 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:50.744642 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:51.124315 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:51.124223 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:51.224727 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:51.224690 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:49:58.227485 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:49:58.227451 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:50:26.651977 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:26.651913 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:29.526830 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:29.526799 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:50:31.195529 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:31.195499 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:31.313210 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:31.313175 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:31.549077 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:31.548953 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:31.801689 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:31.801587 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:31.898583 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:31.898538 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:32.149278 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:32.149242 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:32.507667 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:32.507570 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:32.689317 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:32.689258 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:34.818781 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:34.818748 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:35.059929 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:35.059886 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:35.125163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:35.125069 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:35.743540 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:35.743509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:35.845012 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:35.844977 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:50:43.139124 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:50:43.139080 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:51:10.359009 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:10.358976 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:13.036545 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:13.036510 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:51:14.944708 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:14.944675 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:15.121112 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:15.121080 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:15.365536 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:15.365424 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:15.616476 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:15.616372 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:15.942969 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:15.942926 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:16.116585 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:16.116547 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:17.922986 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:17.922944 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:18.021159 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:18.021124 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:18.209551 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:18.209476 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:18.427239 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:18.427204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:18.993327 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:18.993281 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:19.134523 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:19.134492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:24.523128 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:24.523095 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:51:54.360754 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:54.360718 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:51:57.237362 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:57.237328 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:51:58.904351 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:51:58.904316 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:00.115947 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:00.115918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:00.344775 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:00.344736 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:00.575259 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:00.575225 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:00.921016 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:00.920967 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:01.512533 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:01.512497 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:03.234078 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:03.234027 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:03.342108 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:03.342077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:03.506627 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:03.506535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:03.606238 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:03.606201 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:04.000731 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:04.000698 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:04.100995 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:04.100960 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:08.134664 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:08.134632 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:52:36.256866 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:36.256821 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:36.357160 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:36.357110 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:38.877241 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:38.877193 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:38.983399 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:38.983339 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:39.183376 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:39.183338 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:40.739327 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:40.739294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:42.236438 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:42.236378 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:52:43.186450 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:43.186413 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:43.407639 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:43.407603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:43.752915 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:43.752880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:44.447639 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:44.447606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:44.597091 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:44.597056 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:44.785353 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:44.785261 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:46.757794 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:46.757746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:53.772056 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:53.772019 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:53.867018 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:53.866977 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:53.965940 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:53.965904 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:54.064792 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:54.064693 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:52:54.516370 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:52:54.516329 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:20.508006 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:20.507930 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:26.895019 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:26.894964 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:27.005669 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:27.005627 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:27.096512 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:27.096482 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:43.046214 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:43.046181 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:48.765198 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:48.765165 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:48.835716 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:48.835641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:48.917931 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:48.917891 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:49.016732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:49.016641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:53:59.503987 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:53:59.503936 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:00.366312 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:00.366276 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:07.107780 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:07.107736 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:07.209051 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:07.209009 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:07.308193 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:07.308162 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:07.412993 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:07.412951 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:08.281732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:08.281698 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:08.382165 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:08.382135 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:10.438244 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:10.438208 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:54:11.061132 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:11.061096 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:17.376246 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:17.376202 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:17.488495 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:17.488461 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:18.076106 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:18.076072 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:20.547925 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:20.547887 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:54:24.472000 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:24.471968 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:24.555321 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:24.555277 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:24.756706 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:24.756576 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:25.094852 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:25.094770 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:25.633619 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:25.633588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:25.749546 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:25.749515 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:25.860885 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:25.860852 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:28.522181 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:28.522110 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:28.616571 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:28.616538 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:28.697727 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:28.697687 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:28.796423 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:28.796308 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:29.247004 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:29.246971 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:29.357763 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:29.357733 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:33.045557 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:33.045516 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:33.239834 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:33.239806 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:54:33.714773 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:33.714742 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:33.814864 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:33.814826 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:34.168541 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:34.168505 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:40.054884 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:40.054801 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:40.157353 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:40.157318 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:44.041448 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:44.041401 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:50.937534 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:50.937499 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:54.222914 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:54.222877 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:54:55.503095 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:55.503055 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:54:55.921563 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:54:55.921524 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:01.897585 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:01.897544 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:01.999868 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:01.999826 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:02.199016 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:02.198930 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:02.408670 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:02.408634 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:02.667349 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:02.667315 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:02.767206 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:02.767175 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:02.925162 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:02.925078 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:08.765509 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:08.765474 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:08.832029 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:08.831995 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:08.915942 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:08.915910 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:09.018725 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:09.018635 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:09.152161 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:09.152127 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:14.990928 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:14.990891 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:15.416562 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:15.416529 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:21.909995 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:21.909954 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:22.233887 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:22.233792 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:22.497911 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:22.497821 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:24.849192 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:24.849158 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:24.973876 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:24.973840 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:25.127137 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:25.127035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:25.748153 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:25.748117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:25.895033 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:25.894992 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:29.380141 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:29.380099 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:31.448959 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:31.448922 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:55:36.138284 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:36.138243 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:55:36.237497 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:55:36.237468 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:56:03.440969 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.440938 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6"] Jul 21 08:56:03.441509 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.441264 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbb5a7f-9cba-43b0-a516-7c48b139be6b" containerName="main" Jul 21 08:56:03.441509 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.441274 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbb5a7f-9cba-43b0-a516-7c48b139be6b" containerName="main" Jul 21 08:56:03.441509 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.441287 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c64fea-1604-4ee1-badc-eae9cdcf267b" containerName="main" Jul 21 08:56:03.441509 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.441295 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c64fea-1604-4ee1-badc-eae9cdcf267b" containerName="main" Jul 21 08:56:03.441509 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.441374 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbb5a7f-9cba-43b0-a516-7c48b139be6b" containerName="main" Jul 21 08:56:03.441509 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.441405 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c64fea-1604-4ee1-badc-eae9cdcf267b" containerName="main" Jul 21 08:56:03.444449 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.444432 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.448156 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.448131 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f491e220\"/\"openshift-service-ca.crt\"" Jul 21 08:56:03.448293 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.448245 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f491e220\"/\"kube-root-ca.crt\"" Jul 21 08:56:03.449206 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.449184 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f491e220\"/\"e2e-lifecycle-model-f491e220-kserve-self-signed-certs\"" Jul 21 08:56:03.449295 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.449210 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f491e220\"/\"default-dockercfg-grrzk\"" Jul 21 08:56:03.485592 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.485561 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6"] Jul 21 08:56:03.497593 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.497555 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:56:03.586397 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.586350 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tls-certs\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.586558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.586422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-home\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.586558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.586464 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-model-cache\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.586558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.586487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tmp-dir\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.586558 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.586529 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-dshm\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.586720 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.586574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krb56\" (UniqueName: \"kubernetes.io/projected/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-kube-api-access-krb56\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.687780 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.687742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-model-cache\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.687918 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.687789 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tmp-dir\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.687918 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.687805 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-dshm\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.687918 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.687838 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krb56\" (UniqueName: \"kubernetes.io/projected/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-kube-api-access-krb56\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.687918 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.687890 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tls-certs\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.688091 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.687959 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-home\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.688198 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.688176 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-model-cache\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.688261 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.688242 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tmp-dir\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.688334 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.688319 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-home\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.690126 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.690105 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-dshm\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.690498 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.690482 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tls-certs\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.705854 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.705800 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krb56\" (UniqueName: \"kubernetes.io/projected/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-kube-api-access-krb56\") pod \"e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.753470 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.753443 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:03.893863 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:03.893839 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6"] Jul 21 08:56:03.895604 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:56:03.895574 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6310ede_ab5d_4fe8_b41e_4d343a7902d8.slice/crio-c37505ef3eff91862d2b7eda312fede93d93bd5551b815cec501a416bd0fe038 WatchSource:0}: Error finding container c37505ef3eff91862d2b7eda312fede93d93bd5551b815cec501a416bd0fe038: Status 404 returned error can't find the container with id c37505ef3eff91862d2b7eda312fede93d93bd5551b815cec501a416bd0fe038 Jul 21 08:56:04.834795 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:04.834758 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" event={"ID":"a6310ede-ab5d-4fe8-b41e-4d343a7902d8","Type":"ContainerStarted","Data":"767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227"} Jul 21 08:56:04.834795 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:04.834805 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" event={"ID":"a6310ede-ab5d-4fe8-b41e-4d343a7902d8","Type":"ContainerStarted","Data":"c37505ef3eff91862d2b7eda312fede93d93bd5551b815cec501a416bd0fe038"} Jul 21 08:56:04.851911 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:04.851862 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" podStartSLOduration=1.8518467539999999 podStartE2EDuration="1.851846754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:56:04.850202535 +0000 UTC m=+2236.929240870" watchObservedRunningTime="2026-07-21 08:56:04.851846754 +0000 UTC m=+2236.930885085" Jul 21 08:56:06.628655 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:06.628620 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:56:13.754659 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:13.754578 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:13.754659 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:13.754638 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:13.766890 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:13.766866 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:13.892269 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:13.892243 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:56:20.434577 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:20.434541 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:56:37.934375 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:37.934340 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:56:39.240664 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:39.240628 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:56:43.709693 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:43.709655 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:56:46.067775 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:46.067737 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:56:46.332866 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:46.332783 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:56:51.050195 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:51.050157 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6"] Jul 21 08:56:51.145897 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:56:51.145859 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:06.072424 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.072359 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" podUID="a6310ede-ab5d-4fe8-b41e-4d343a7902d8" containerName="main" containerID="cri-o://767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227" gracePeriod=45 Jul 21 08:57:06.314183 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.314160 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:57:06.417802 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.417778 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krb56\" (UniqueName: \"kubernetes.io/projected/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-kube-api-access-krb56\") pod \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " Jul 21 08:57:06.417978 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.417821 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-home\") pod \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " Jul 21 08:57:06.417978 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.417860 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tls-certs\") pod \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " Jul 21 08:57:06.417978 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.417898 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-dshm\") pod \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " Jul 21 08:57:06.418117 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.417979 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-model-cache\") pod \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " Jul 21 08:57:06.418117 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.418017 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tmp-dir\") pod \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\" (UID: \"a6310ede-ab5d-4fe8-b41e-4d343a7902d8\") " Jul 21 08:57:06.418117 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.418087 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-home" (OuterVolumeSpecName: "home") pod "a6310ede-ab5d-4fe8-b41e-4d343a7902d8" (UID: "a6310ede-ab5d-4fe8-b41e-4d343a7902d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:57:06.418289 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.418236 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6310ede-ab5d-4fe8-b41e-4d343a7902d8" (UID: "a6310ede-ab5d-4fe8-b41e-4d343a7902d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:57:06.418441 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.418318 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6310ede-ab5d-4fe8-b41e-4d343a7902d8" (UID: "a6310ede-ab5d-4fe8-b41e-4d343a7902d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:57:06.418441 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.418331 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:57:06.420248 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.420224 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-dshm" (OuterVolumeSpecName: "dshm") pod "a6310ede-ab5d-4fe8-b41e-4d343a7902d8" (UID: "a6310ede-ab5d-4fe8-b41e-4d343a7902d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:57:06.420344 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.420285 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6310ede-ab5d-4fe8-b41e-4d343a7902d8" (UID: "a6310ede-ab5d-4fe8-b41e-4d343a7902d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:57:06.420344 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.420302 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-kube-api-access-krb56" (OuterVolumeSpecName: "kube-api-access-krb56") pod "a6310ede-ab5d-4fe8-b41e-4d343a7902d8" (UID: "a6310ede-ab5d-4fe8-b41e-4d343a7902d8"). InnerVolumeSpecName "kube-api-access-krb56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:57:06.519428 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.519374 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:57:06.519428 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.519424 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:57:06.519428 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.519433 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:57:06.519650 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.519462 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:57:06.519650 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:06.519471 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krb56\" (UniqueName: \"kubernetes.io/projected/a6310ede-ab5d-4fe8-b41e-4d343a7902d8-kube-api-access-krb56\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 08:57:07.158995 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:07.158960 2563 generic.go:356] "Generic (PLEG): container finished" podID="a6310ede-ab5d-4fe8-b41e-4d343a7902d8" containerID="767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227" exitCode=0 Jul 21 08:57:07.159519 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:07.159002 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" Jul 21 08:57:07.159519 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:07.159036 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" event={"ID":"a6310ede-ab5d-4fe8-b41e-4d343a7902d8","Type":"ContainerDied","Data":"767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227"} Jul 21 08:57:07.159519 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:07.159084 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6" event={"ID":"a6310ede-ab5d-4fe8-b41e-4d343a7902d8","Type":"ContainerDied","Data":"c37505ef3eff91862d2b7eda312fede93d93bd5551b815cec501a416bd0fe038"} Jul 21 08:57:07.159519 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:07.159109 2563 scope.go:117] "RemoveContainer" containerID="767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227" Jul 21 08:57:07.168602 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:07.168586 2563 scope.go:117] "RemoveContainer" containerID="767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227" Jul 21 08:57:07.168869 ip-10-0-137-36 kubenswrapper[2563]: E0721 08:57:07.168850 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227\": container with ID starting with 767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227 not found: ID does not exist" containerID="767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227" Jul 21 08:57:07.168931 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:07.168883 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227"} err="failed to get container status \"767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227\": rpc error: code = NotFound desc = could not find container \"767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227\": container with ID starting with 767ad8eb45e0e78165817ae9797d393c4159a7e3f31d7d0bfd5a5102d1e40227 not found: ID does not exist" Jul 21 08:57:07.180662 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:07.180640 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6"] Jul 21 08:57:07.184525 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:07.184506 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f491e220/e2e-lifecycle-model-f491e220-kserve-78df7496c9-frjm6"] Jul 21 08:57:08.358001 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:08.357971 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6310ede-ab5d-4fe8-b41e-4d343a7902d8" path="/var/lib/kubelet/pods/a6310ede-ab5d-4fe8-b41e-4d343a7902d8/volumes" Jul 21 08:57:27.193922 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:27.193877 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:28.185968 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:28.185934 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:33.700854 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:33.700815 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:33.800898 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:33.800865 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:34.073293 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:34.073205 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:34.269139 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:34.269104 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:35.385163 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:35.385072 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:35.488973 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:35.488940 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:35.587070 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:35.587031 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:36.380433 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:36.380372 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:36.477749 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:36.477713 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:36.602801 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:36.602775 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:37.230960 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:37.230930 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:37.330428 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:37.330372 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:37.834039 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:37.833998 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:57:38.015903 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:38.015868 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:44.319896 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:44.319802 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:44.442710 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:44.442669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:45.099469 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:45.099432 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:48.379986 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:48.379936 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:57:51.845646 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:51.845605 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:52.052560 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:52.052523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:57:52.461606 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:57:52.461573 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:02.650569 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:02.650529 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:03.005937 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:03.005827 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:23.998143 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:23.998109 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:32.016777 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:32.016727 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:32.164454 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:32.164423 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:32.422637 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:32.422602 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:32.899527 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:32.899493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:33.263060 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:33.262957 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:36.111167 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:36.111136 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:36.307862 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:36.307826 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:36.444081 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:36.444053 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:37.045034 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:37.044996 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:37.458840 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:37.458795 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:37.623789 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:37.623746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:39.598432 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:39.598402 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:58:40.593763 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:40.593721 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:40.993299 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:40.993268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:41.470652 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:41.470621 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:41.638580 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:41.638546 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:46.916327 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:46.916294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:47.037305 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:47.037264 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:47.451928 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:47.451889 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:47.668570 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:47.668537 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:48.147724 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:48.147693 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:48.249061 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:48.249024 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:50.666487 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:50.666454 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:58:51.364340 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:51.364310 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:58.692372 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:58.692336 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:58:58.799961 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:58:58.799912 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:59:07.960722 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.960687 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw"] Jul 21 08:59:07.961095 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.961060 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6310ede-ab5d-4fe8-b41e-4d343a7902d8" containerName="main" Jul 21 08:59:07.961095 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.961072 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6310ede-ab5d-4fe8-b41e-4d343a7902d8" containerName="main" Jul 21 08:59:07.961171 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.961132 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6310ede-ab5d-4fe8-b41e-4d343a7902d8" containerName="main" Jul 21 08:59:07.964168 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.964153 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:07.966744 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.966715 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8d885f\"/\"default-dockercfg-2n7wh\"" Jul 21 08:59:07.967650 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.967628 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8d885f\"/\"kube-root-ca.crt\"" Jul 21 08:59:07.967783 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.967628 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8d885f\"/\"test-model-8d885f-kserve-self-signed-certs\"" Jul 21 08:59:07.967946 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.967930 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8d885f\"/\"openshift-service-ca.crt\"" Jul 21 08:59:07.983802 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:07.983781 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw"] Jul 21 08:59:08.009236 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.009193 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:59:08.092561 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.092532 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-model-cache\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.092732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.092564 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxql\" (UniqueName: \"kubernetes.io/projected/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-kube-api-access-dgxql\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.092732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.092619 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-dshm\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.092732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.092642 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tls-certs\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.092732 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.092658 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tmp-dir\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.092922 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.092757 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-home\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.193573 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.193538 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-dshm\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.193764 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.193577 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tls-certs\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.193764 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.193602 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tmp-dir\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.193764 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.193638 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-home\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.193764 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.193730 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-model-cache\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.193764 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.193753 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxql\" (UniqueName: \"kubernetes.io/projected/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-kube-api-access-dgxql\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.194087 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.194051 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tmp-dir\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.194159 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.194101 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-home\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.194159 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.194117 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-model-cache\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.196036 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.196009 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-dshm\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.196328 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.196311 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tls-certs\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.208846 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.208825 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxql\" (UniqueName: \"kubernetes.io/projected/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-kube-api-access-dgxql\") pod \"test-model-8d885f-kserve-cd86648c-9bgbw\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.273067 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.272984 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:08.417670 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.417646 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw"] Jul 21 08:59:08.420128 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:59:08.420103 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca0e2ffe_5c59_4c6f_bf57_990d0636fe56.slice/crio-bbf0a466683549cec6effbc7009f122f7d4523e03c4b6e8c900d73ef7be583d3 WatchSource:0}: Error finding container bbf0a466683549cec6effbc7009f122f7d4523e03c4b6e8c900d73ef7be583d3: Status 404 returned error can't find the container with id bbf0a466683549cec6effbc7009f122f7d4523e03c4b6e8c900d73ef7be583d3 Jul 21 08:59:08.789583 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.789542 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" event={"ID":"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56","Type":"ContainerStarted","Data":"d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb"} Jul 21 08:59:08.789583 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.789589 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" event={"ID":"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56","Type":"ContainerStarted","Data":"bbf0a466683549cec6effbc7009f122f7d4523e03c4b6e8c900d73ef7be583d3"} Jul 21 08:59:08.811918 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:08.811868 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" podStartSLOduration=1.811853015 podStartE2EDuration="1.811853015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:59:08.810528841 +0000 UTC m=+2420.889567173" watchObservedRunningTime="2026-07-21 08:59:08.811853015 +0000 UTC m=+2420.890891346" Jul 21 08:59:10.645652 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:10.645621 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:59:10.850849 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:10.850806 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:59:11.177956 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:11.177920 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:59:18.273635 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:18.273601 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:18.273635 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:18.273645 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:18.286127 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:18.286098 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:18.851229 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:18.851197 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 08:59:23.267164 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:23.267126 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:59:35.963652 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:35.963616 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm"] Jul 21 08:59:35.966911 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:35.966895 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:35.970142 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:35.970121 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c941d0\"/\"kube-root-ca.crt\"" Jul 21 08:59:35.970282 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:35.970154 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c941d0\"/\"default-dockercfg-hm9gq\"" Jul 21 08:59:35.970969 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:35.970948 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c941d0\"/\"test-model-c941d0-kserve-self-signed-certs\"" Jul 21 08:59:35.971084 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:35.970980 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c941d0\"/\"openshift-service-ca.crt\"" Jul 21 08:59:35.985642 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:35.985622 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm"] Jul 21 08:59:36.013240 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.013200 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:59:36.033919 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.033889 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-home\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.033919 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.033918 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tls-certs\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.034092 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.033947 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-model-cache\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.034092 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.033963 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-dshm\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.034092 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.034031 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tmp-dir\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.034092 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.034057 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwgb\" (UniqueName: \"kubernetes.io/projected/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-kube-api-access-mlwgb\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.134964 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.134931 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-model-cache\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.134964 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.134966 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-dshm\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.135274 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.134999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tmp-dir\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.135274 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.135023 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwgb\" (UniqueName: \"kubernetes.io/projected/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-kube-api-access-mlwgb\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.135274 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.135121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-home\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.135274 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.135140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tls-certs\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.135528 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.135513 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-model-cache\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.135576 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.135546 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tmp-dir\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.135612 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.135581 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-home\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.137336 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.137319 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-dshm\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.137659 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.137643 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tls-certs\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.147461 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.147441 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwgb\" (UniqueName: \"kubernetes.io/projected/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-kube-api-access-mlwgb\") pod \"test-model-c941d0-kserve-b754f687f-nkcsm\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.277185 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.277104 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:36.419981 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.419958 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm"] Jul 21 08:59:36.422079 ip-10-0-137-36 kubenswrapper[2563]: W0721 08:59:36.422051 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b80cf87_dcf8_47d6_8cee_caa9116fc7c3.slice/crio-ed8db57e72e591158133103d526b9c84e9877ae766a4303a9d89a20fd1cf8e47 WatchSource:0}: Error finding container ed8db57e72e591158133103d526b9c84e9877ae766a4303a9d89a20fd1cf8e47: Status 404 returned error can't find the container with id ed8db57e72e591158133103d526b9c84e9877ae766a4303a9d89a20fd1cf8e47 Jul 21 08:59:36.933409 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.933349 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" event={"ID":"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3","Type":"ContainerStarted","Data":"acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184"} Jul 21 08:59:36.933409 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.933405 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" event={"ID":"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3","Type":"ContainerStarted","Data":"ed8db57e72e591158133103d526b9c84e9877ae766a4303a9d89a20fd1cf8e47"} Jul 21 08:59:36.955116 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:36.955063 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" podStartSLOduration=1.955047343 podStartE2EDuration="1.955047343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:59:36.954649664 +0000 UTC m=+2449.033687996" watchObservedRunningTime="2026-07-21 08:59:36.955047343 +0000 UTC m=+2449.034085674" Jul 21 08:59:40.363568 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:40.363538 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 08:59:43.728444 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:43.728408 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:59:43.828239 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:43.828204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 08:59:46.278239 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:46.278210 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:46.278630 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:46.278500 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:46.290929 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:46.290908 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:46.993576 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:46.993545 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 08:59:54.443371 ip-10-0-137-36 kubenswrapper[2563]: I0721 08:59:54.443339 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:00:16.479636 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:16.479604 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw"] Jul 21 09:00:16.502444 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:16.502370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:17.416526 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:17.416497 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm"] Jul 21 09:00:17.444348 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:17.444306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:18.257227 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:18.257197 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:18.366448 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:18.366415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:21.634105 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:21.634064 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:00:28.208551 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:28.208509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:31.500772 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.500735 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" podUID="ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" containerName="main" containerID="cri-o://d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb" gracePeriod=45 Jul 21 09:00:31.753287 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.753232 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 09:00:31.847518 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.847480 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tls-certs\") pod \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " Jul 21 09:00:31.847701 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.847534 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-dshm\") pod \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " Jul 21 09:00:31.847701 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.847564 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-model-cache\") pod \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " Jul 21 09:00:31.847701 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.847590 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgxql\" (UniqueName: \"kubernetes.io/projected/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-kube-api-access-dgxql\") pod \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " Jul 21 09:00:31.847701 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.847672 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tmp-dir\") pod \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " Jul 21 09:00:31.847917 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.847755 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-home\") pod \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\" (UID: \"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56\") " Jul 21 09:00:31.847917 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.847820 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" (UID: "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:31.848017 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.847947 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" (UID: "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:31.848017 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.847992 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-home" (OuterVolumeSpecName: "home") pod "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" (UID: "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:31.848106 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.848090 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:31.848142 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.848112 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:31.848142 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.848124 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:31.849901 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.849880 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-dshm" (OuterVolumeSpecName: "dshm") pod "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" (UID: "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:31.850334 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.850302 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" (UID: "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:00:31.850399 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.850313 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-kube-api-access-dgxql" (OuterVolumeSpecName: "kube-api-access-dgxql") pod "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" (UID: "ca0e2ffe-5c59-4c6f-bf57-990d0636fe56"). InnerVolumeSpecName "kube-api-access-dgxql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:00:31.948718 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.948685 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:31.948718 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.948709 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:31.948718 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:31.948720 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgxql\" (UniqueName: \"kubernetes.io/projected/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56-kube-api-access-dgxql\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:32.211006 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.210970 2563 generic.go:356] "Generic (PLEG): container finished" podID="ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" containerID="d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb" exitCode=0 Jul 21 09:00:32.211215 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.211025 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" event={"ID":"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56","Type":"ContainerDied","Data":"d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb"} Jul 21 09:00:32.211215 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.211028 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" Jul 21 09:00:32.211215 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.211056 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw" event={"ID":"ca0e2ffe-5c59-4c6f-bf57-990d0636fe56","Type":"ContainerDied","Data":"bbf0a466683549cec6effbc7009f122f7d4523e03c4b6e8c900d73ef7be583d3"} Jul 21 09:00:32.211215 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.211081 2563 scope.go:117] "RemoveContainer" containerID="d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb" Jul 21 09:00:32.220540 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.220486 2563 scope.go:117] "RemoveContainer" containerID="d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb" Jul 21 09:00:32.220753 ip-10-0-137-36 kubenswrapper[2563]: E0721 09:00:32.220735 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb\": container with ID starting with d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb not found: ID does not exist" containerID="d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb" Jul 21 09:00:32.220817 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.220767 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb"} err="failed to get container status \"d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb\": rpc error: code = NotFound desc = could not find container \"d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb\": container with ID starting with d9521fe862c2516ff6c2be918f2937caef3c4dbb12e37683a08c420c9e280eeb not found: ID does not exist" Jul 21 09:00:32.249464 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.249434 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw"] Jul 21 09:00:32.256911 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.256891 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8d885f/test-model-8d885f-kserve-cd86648c-9bgbw"] Jul 21 09:00:32.358905 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.358878 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" path="/var/lib/kubelet/pods/ca0e2ffe-5c59-4c6f-bf57-990d0636fe56/volumes" Jul 21 09:00:32.437731 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.437689 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" podUID="0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" containerName="main" containerID="cri-o://acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184" gracePeriod=45 Jul 21 09:00:32.688927 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.688904 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 09:00:32.756525 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756499 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tmp-dir\") pod \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " Jul 21 09:00:32.756728 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756555 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tls-certs\") pod \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " Jul 21 09:00:32.756728 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756605 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlwgb\" (UniqueName: \"kubernetes.io/projected/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-kube-api-access-mlwgb\") pod \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " Jul 21 09:00:32.756728 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756670 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-model-cache\") pod \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " Jul 21 09:00:32.756728 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756697 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-home\") pod \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " Jul 21 09:00:32.756968 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756734 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-dshm\") pod \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\" (UID: \"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3\") " Jul 21 09:00:32.756968 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756758 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" (UID: "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:32.757097 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756977 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:32.757097 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756975 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-home" (OuterVolumeSpecName: "home") pod "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" (UID: "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:32.757097 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.756989 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" (UID: "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:32.759152 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.759131 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" (UID: "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:00:32.759431 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.759410 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-kube-api-access-mlwgb" (OuterVolumeSpecName: "kube-api-access-mlwgb") pod "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" (UID: "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3"). InnerVolumeSpecName "kube-api-access-mlwgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:00:32.759431 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.759417 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-dshm" (OuterVolumeSpecName: "dshm") pod "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" (UID: "0b80cf87-dcf8-47d6-8cee-caa9116fc7c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:32.857993 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.857959 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:32.857993 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.857989 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:32.857993 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.858000 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlwgb\" (UniqueName: \"kubernetes.io/projected/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-kube-api-access-mlwgb\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:32.858223 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.858010 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:32.858223 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.858018 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:00:32.948481 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:32.948454 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:33.212739 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.212697 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:33.217593 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.217568 2563 generic.go:356] "Generic (PLEG): container finished" podID="0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" containerID="acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184" exitCode=0 Jul 21 09:00:33.217717 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.217612 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" Jul 21 09:00:33.217717 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.217656 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" event={"ID":"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3","Type":"ContainerDied","Data":"acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184"} Jul 21 09:00:33.217717 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.217702 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm" event={"ID":"0b80cf87-dcf8-47d6-8cee-caa9116fc7c3","Type":"ContainerDied","Data":"ed8db57e72e591158133103d526b9c84e9877ae766a4303a9d89a20fd1cf8e47"} Jul 21 09:00:33.217873 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.217721 2563 scope.go:117] "RemoveContainer" containerID="acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184" Jul 21 09:00:33.227252 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.227234 2563 scope.go:117] "RemoveContainer" containerID="acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184" Jul 21 09:00:33.227530 ip-10-0-137-36 kubenswrapper[2563]: E0721 09:00:33.227511 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184\": container with ID starting with acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184 not found: ID does not exist" containerID="acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184" Jul 21 09:00:33.227603 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.227540 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184"} err="failed to get container status \"acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184\": rpc error: code = NotFound desc = could not find container \"acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184\": container with ID starting with acb9e016836e6b106a5ae5f949f270b84ec0c664a75ac0e8cd0ee0b147ff0184 not found: ID does not exist" Jul 21 09:00:33.245237 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.245213 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm"] Jul 21 09:00:33.252970 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:33.252950 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c941d0/test-model-c941d0-kserve-b754f687f-nkcsm"] Jul 21 09:00:34.056235 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:34.056199 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:00:34.358059 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:34.357980 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" path="/var/lib/kubelet/pods/0b80cf87-dcf8-47d6-8cee-caa9116fc7c3/volumes" Jul 21 09:00:39.263193 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:39.263118 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:39.384991 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:39.384965 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:39.521497 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:39.521409 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:39.742636 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:39.742606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:40.673038 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:40.673003 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:40.790139 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:40.790089 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:48.370565 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:48.370533 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:50.490794 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:50.490758 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:53.894451 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:53.894411 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:00:54.585048 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:54.585016 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:54.970886 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:54.970852 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:00:55.379279 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:00:55.379153 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:01.880578 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:01.880528 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:01.963608 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:01.963564 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:02.056315 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:02.056281 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:02.254150 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:02.254113 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:02.636118 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:02.636033 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:03.004654 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:03.004609 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:03.257742 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:03.257651 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:05.704744 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:05.704705 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:05.804572 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:05.804533 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:06.132158 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:06.132071 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:06.545762 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:06.545720 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:06.840659 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:06.840561 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:06.935059 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:06.935018 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:10.203087 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:10.203052 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:10.702515 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:10.702479 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:11.063700 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:11.063608 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:11.580186 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:11.580155 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:13.037996 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:13.037964 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:01:13.715108 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:13.715069 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:13.803995 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:13.803966 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:13.991993 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:13.991908 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:14.172515 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:14.172486 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:14.932057 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:14.932022 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:15.101679 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:15.101646 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:18.254484 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:18.254446 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:26.997649 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:26.997611 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:27.099478 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:27.099442 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:30.648517 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:30.648476 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:01:37.938032 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:37.937993 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:38.038135 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:38.038095 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:43.243715 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:43.243679 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:46.635763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:46.635728 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:01:47.543765 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:47.543731 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:54.859588 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:54.859532 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:55.010857 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:55.010824 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:55.237523 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:55.237489 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:55.486536 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:55.486510 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:01:55.645855 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:01:55.645822 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:11.287472 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:11.287376 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:15.831576 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:15.831537 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:16.031620 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:16.031581 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:02:16.213484 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:16.213427 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:22.259283 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:22.259232 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:22.341730 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:22.341698 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:22.653321 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:22.653287 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:22.926600 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:22.926507 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:23.281321 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:23.281225 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:23.719084 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:23.719051 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:23.988831 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:23.988727 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:27.233151 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:27.233114 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:27.354339 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:27.354305 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:27.512262 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:27.512171 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:27.900440 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:27.900409 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:28.360686 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:28.360602 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:28.477370 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:28.477339 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:33.065223 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:33.065184 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:33.313701 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:33.313669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:35.081587 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:35.081544 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:02:35.228302 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:35.228269 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:35.361668 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:35.361576 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:35.663601 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:35.663565 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:36.374798 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:36.374765 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:36.495056 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:36.495021 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:38.348013 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:38.347979 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:38.473324 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:38.473286 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:39.356058 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:39.356022 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:39.459010 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:39.458971 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:39.612573 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:39.612465 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:47.160082 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:47.160042 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:02:50.471678 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:50.471639 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:50.577403 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:50.577350 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:56.586401 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.586355 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw"] Jul 21 09:02:56.586763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.586702 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" containerName="main" Jul 21 09:02:56.586763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.586714 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" containerName="main" Jul 21 09:02:56.586763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.586728 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" containerName="main" Jul 21 09:02:56.586763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.586733 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" containerName="main" Jul 21 09:02:56.586896 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.586787 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca0e2ffe-5c59-4c6f-bf57-990d0636fe56" containerName="main" Jul 21 09:02:56.586896 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.586796 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b80cf87-dcf8-47d6-8cee-caa9116fc7c3" containerName="main" Jul 21 09:02:56.589626 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.589608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.594154 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.594126 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-824408\"/\"default-dockercfg-n8xwq\"" Jul 21 09:02:56.594154 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.594138 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-824408\"/\"openshift-service-ca.crt\"" Jul 21 09:02:56.594154 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.594128 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-824408\"/\"kube-root-ca.crt\"" Jul 21 09:02:56.594420 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.594188 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-824408\"/\"auth-test-model-824408-kserve-self-signed-certs\"" Jul 21 09:02:56.597913 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.597892 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-home\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.598012 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.597932 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10b5ab7-a357-445f-ba3c-32e716496886-tls-certs\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.598012 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.597976 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-model-cache\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.598012 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.598002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-dshm\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.598159 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.598020 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-tmp-dir\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.598159 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.598044 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c22ts\" (UniqueName: \"kubernetes.io/projected/b10b5ab7-a357-445f-ba3c-32e716496886-kube-api-access-c22ts\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.621647 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.621618 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw"] Jul 21 09:02:56.647194 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.647135 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:02:56.698745 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.698715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-dshm\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.698903 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.698749 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-tmp-dir\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.698903 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.698771 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c22ts\" (UniqueName: \"kubernetes.io/projected/b10b5ab7-a357-445f-ba3c-32e716496886-kube-api-access-c22ts\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.698903 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.698841 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-home\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.698903 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.698892 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10b5ab7-a357-445f-ba3c-32e716496886-tls-certs\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.699126 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.698921 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-model-cache\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.699225 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.699205 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-tmp-dir\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.699332 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.699298 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-model-cache\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.699426 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.699346 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-home\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.701000 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.700983 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-dshm\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.701397 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.701360 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10b5ab7-a357-445f-ba3c-32e716496886-tls-certs\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.708783 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.708762 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c22ts\" (UniqueName: \"kubernetes.io/projected/b10b5ab7-a357-445f-ba3c-32e716496886-kube-api-access-c22ts\") pod \"auth-test-model-824408-kserve-558659-znslw\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:56.899016 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:56.898978 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:02:57.046820 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:57.046792 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw"] Jul 21 09:02:57.048316 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:02:57.048289 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb10b5ab7_a357_445f_ba3c_32e716496886.slice/crio-355cceb95d550c27aabcefa80c6ea104ee1ce777076529f4e0b57b65b45a6ef0 WatchSource:0}: Error finding container 355cceb95d550c27aabcefa80c6ea104ee1ce777076529f4e0b57b65b45a6ef0: Status 404 returned error can't find the container with id 355cceb95d550c27aabcefa80c6ea104ee1ce777076529f4e0b57b65b45a6ef0 Jul 21 09:02:57.972658 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:57.972623 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" event={"ID":"b10b5ab7-a357-445f-ba3c-32e716496886","Type":"ContainerStarted","Data":"a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3"} Jul 21 09:02:57.972658 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:57.972670 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" event={"ID":"b10b5ab7-a357-445f-ba3c-32e716496886","Type":"ContainerStarted","Data":"355cceb95d550c27aabcefa80c6ea104ee1ce777076529f4e0b57b65b45a6ef0"} Jul 21 09:02:57.994188 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:57.994141 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" podStartSLOduration=1.9941243119999998 podStartE2EDuration="1.994124312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:02:57.992072138 +0000 UTC m=+2650.071110468" watchObservedRunningTime="2026-07-21 09:02:57.994124312 +0000 UTC m=+2650.073162676" Jul 21 09:02:59.738552 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:02:59.738516 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:03:00.228673 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:00.228637 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:03:00.344149 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:00.344116 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:03:06.899200 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:06.899167 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:03:06.899716 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:06.899218 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:03:06.911702 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:06.911679 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:03:07.028747 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:07.028714 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:03:23.645029 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.644990 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq"] Jul 21 09:03:23.648882 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.648860 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.654662 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.654624 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"kube-root-ca.crt\"" Jul 21 09:03:23.654828 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.654624 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"auth-test-model-26961c-kserve-self-signed-certs\"" Jul 21 09:03:23.654828 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.654722 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"openshift-service-ca.crt\"" Jul 21 09:03:23.654828 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.654772 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"default-dockercfg-xnp7p\"" Jul 21 09:03:23.674759 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.674730 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq"] Jul 21 09:03:23.725806 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.725771 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-dshm\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.725984 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.725809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-tmp-dir\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.725984 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.725884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4hb\" (UniqueName: \"kubernetes.io/projected/ce2e163a-bf72-46e7-8020-de86111d7478-kube-api-access-nf4hb\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.725984 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.725934 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-home\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.725984 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.725974 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2e163a-bf72-46e7-8020-de86111d7478-tls-certs\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.726162 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.726016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-model-cache\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.737079 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.737042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:03:23.827096 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.827066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4hb\" (UniqueName: \"kubernetes.io/projected/ce2e163a-bf72-46e7-8020-de86111d7478-kube-api-access-nf4hb\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.827249 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.827101 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-home\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.827249 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.827143 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2e163a-bf72-46e7-8020-de86111d7478-tls-certs\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.827249 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.827179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-model-cache\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.827413 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.827249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-dshm\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.827413 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.827274 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-tmp-dir\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.827592 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.827568 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-home\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.827664 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.827605 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-model-cache\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.827664 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.827631 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-tmp-dir\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.829593 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.829573 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-dshm\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.829687 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.829672 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2e163a-bf72-46e7-8020-de86111d7478-tls-certs\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.857214 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.857186 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4hb\" (UniqueName: \"kubernetes.io/projected/ce2e163a-bf72-46e7-8020-de86111d7478-kube-api-access-nf4hb\") pod \"auth-test-model-26961c-kserve-849d8bcf67-6j5cq\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:23.958081 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:23.957995 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:24.120842 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:24.120667 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq"] Jul 21 09:03:24.123179 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:03:24.123151 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce2e163a_bf72_46e7_8020_de86111d7478.slice/crio-35f228175e6944af40da6926a721e8c448573c0d4b7e0574c00b6007a3a7c47d WatchSource:0}: Error finding container 35f228175e6944af40da6926a721e8c448573c0d4b7e0574c00b6007a3a7c47d: Status 404 returned error can't find the container with id 35f228175e6944af40da6926a721e8c448573c0d4b7e0574c00b6007a3a7c47d Jul 21 09:03:25.108038 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:25.107993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" event={"ID":"ce2e163a-bf72-46e7-8020-de86111d7478","Type":"ContainerStarted","Data":"bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606"} Jul 21 09:03:25.108466 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:25.108046 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" event={"ID":"ce2e163a-bf72-46e7-8020-de86111d7478","Type":"ContainerStarted","Data":"35f228175e6944af40da6926a721e8c448573c0d4b7e0574c00b6007a3a7c47d"} Jul 21 09:03:25.164428 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:25.164353 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" podStartSLOduration=2.164323262 podStartE2EDuration="2.164323262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:03:25.15229323 +0000 UTC m=+2677.231331560" watchObservedRunningTime="2026-07-21 09:03:25.164323262 +0000 UTC m=+2677.243361593" Jul 21 09:03:27.076799 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:27.076757 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:03:33.958789 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:33.958755 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:33.959295 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:33.959063 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:33.971610 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:33.971583 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:34.167120 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:34.167092 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:03:39.361036 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:39.360945 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:03:39.502882 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:39.502845 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:03:46.373832 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:03:46.373800 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:04:03.499376 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:04:03.499339 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:04:06.331617 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:04:06.331583 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:04:22.238143 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:04:22.238113 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:04:42.081080 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:04:42.081046 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:05:01.841271 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:01.841237 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:05:04.163419 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:04.163361 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:05:04.264872 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:04.264838 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:05:20.934837 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:20.934757 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:05:37.977192 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:37.977153 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw"] Jul 21 09:05:38.027683 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:38.027648 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:05:38.961516 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:38.961487 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq"] Jul 21 09:05:38.995874 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:38.995837 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:05:41.133880 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:41.133846 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:05:52.997349 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:52.997297 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" podUID="b10b5ab7-a357-445f-ba3c-32e716496886" containerName="main" containerID="cri-o://a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3" gracePeriod=45 Jul 21 09:05:53.248858 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.248803 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:05:53.351993 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.351960 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-model-cache\") pod \"b10b5ab7-a357-445f-ba3c-32e716496886\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " Jul 21 09:05:53.352180 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.352002 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-tmp-dir\") pod \"b10b5ab7-a357-445f-ba3c-32e716496886\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " Jul 21 09:05:53.352180 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.352039 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10b5ab7-a357-445f-ba3c-32e716496886-tls-certs\") pod \"b10b5ab7-a357-445f-ba3c-32e716496886\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " Jul 21 09:05:53.352180 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.352056 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-dshm\") pod \"b10b5ab7-a357-445f-ba3c-32e716496886\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " Jul 21 09:05:53.352180 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.352073 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-home\") pod \"b10b5ab7-a357-445f-ba3c-32e716496886\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " Jul 21 09:05:53.352180 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.352169 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c22ts\" (UniqueName: \"kubernetes.io/projected/b10b5ab7-a357-445f-ba3c-32e716496886-kube-api-access-c22ts\") pod \"b10b5ab7-a357-445f-ba3c-32e716496886\" (UID: \"b10b5ab7-a357-445f-ba3c-32e716496886\") " Jul 21 09:05:53.352459 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.352274 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-model-cache" (OuterVolumeSpecName: "model-cache") pod "b10b5ab7-a357-445f-ba3c-32e716496886" (UID: "b10b5ab7-a357-445f-ba3c-32e716496886"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:05:53.352459 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.352415 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b10b5ab7-a357-445f-ba3c-32e716496886" (UID: "b10b5ab7-a357-445f-ba3c-32e716496886"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:05:53.352459 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.352432 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-home" (OuterVolumeSpecName: "home") pod "b10b5ab7-a357-445f-ba3c-32e716496886" (UID: "b10b5ab7-a357-445f-ba3c-32e716496886"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:05:53.352571 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.352441 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:53.354543 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.354520 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-dshm" (OuterVolumeSpecName: "dshm") pod "b10b5ab7-a357-445f-ba3c-32e716496886" (UID: "b10b5ab7-a357-445f-ba3c-32e716496886"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:05:53.354650 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.354534 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10b5ab7-a357-445f-ba3c-32e716496886-kube-api-access-c22ts" (OuterVolumeSpecName: "kube-api-access-c22ts") pod "b10b5ab7-a357-445f-ba3c-32e716496886" (UID: "b10b5ab7-a357-445f-ba3c-32e716496886"). InnerVolumeSpecName "kube-api-access-c22ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:05:53.354650 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.354550 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10b5ab7-a357-445f-ba3c-32e716496886-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b10b5ab7-a357-445f-ba3c-32e716496886" (UID: "b10b5ab7-a357-445f-ba3c-32e716496886"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:05:53.453772 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.453737 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:53.453772 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.453764 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10b5ab7-a357-445f-ba3c-32e716496886-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:53.453772 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.453773 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:53.453998 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.453781 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10b5ab7-a357-445f-ba3c-32e716496886-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:53.453998 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.453790 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c22ts\" (UniqueName: \"kubernetes.io/projected/b10b5ab7-a357-445f-ba3c-32e716496886-kube-api-access-c22ts\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:53.884024 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.883988 2563 generic.go:356] "Generic (PLEG): container finished" podID="b10b5ab7-a357-445f-ba3c-32e716496886" containerID="a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3" exitCode=0 Jul 21 09:05:53.884024 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.884023 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" Jul 21 09:05:53.884245 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.884051 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" event={"ID":"b10b5ab7-a357-445f-ba3c-32e716496886","Type":"ContainerDied","Data":"a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3"} Jul 21 09:05:53.884245 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.884093 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw" event={"ID":"b10b5ab7-a357-445f-ba3c-32e716496886","Type":"ContainerDied","Data":"355cceb95d550c27aabcefa80c6ea104ee1ce777076529f4e0b57b65b45a6ef0"} Jul 21 09:05:53.884245 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.884111 2563 scope.go:117] "RemoveContainer" containerID="a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3" Jul 21 09:05:53.893873 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.893782 2563 scope.go:117] "RemoveContainer" containerID="a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3" Jul 21 09:05:53.894115 ip-10-0-137-36 kubenswrapper[2563]: E0721 09:05:53.894100 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3\": container with ID starting with a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3 not found: ID does not exist" containerID="a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3" Jul 21 09:05:53.894173 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.894121 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3"} err="failed to get container status \"a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3\": rpc error: code = NotFound desc = could not find container \"a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3\": container with ID starting with a2dfeeb90e622417adda2b742743424bea941496b072f6aac4110edcf1a84ca3 not found: ID does not exist" Jul 21 09:05:53.906033 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.906012 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw"] Jul 21 09:05:53.911823 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.911802 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-824408/auth-test-model-824408-kserve-558659-znslw"] Jul 21 09:05:53.981870 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:53.981830 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" podUID="ce2e163a-bf72-46e7-8020-de86111d7478" containerName="main" containerID="cri-o://bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606" gracePeriod=45 Jul 21 09:05:54.235416 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.235396 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:05:54.357872 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.357839 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b10b5ab7-a357-445f-ba3c-32e716496886" path="/var/lib/kubelet/pods/b10b5ab7-a357-445f-ba3c-32e716496886/volumes" Jul 21 09:05:54.361076 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361053 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-home\") pod \"ce2e163a-bf72-46e7-8020-de86111d7478\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " Jul 21 09:05:54.361215 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361102 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-model-cache\") pod \"ce2e163a-bf72-46e7-8020-de86111d7478\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " Jul 21 09:05:54.361215 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361192 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-tmp-dir\") pod \"ce2e163a-bf72-46e7-8020-de86111d7478\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " Jul 21 09:05:54.361315 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361238 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf4hb\" (UniqueName: \"kubernetes.io/projected/ce2e163a-bf72-46e7-8020-de86111d7478-kube-api-access-nf4hb\") pod \"ce2e163a-bf72-46e7-8020-de86111d7478\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " Jul 21 09:05:54.361315 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361288 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2e163a-bf72-46e7-8020-de86111d7478-tls-certs\") pod \"ce2e163a-bf72-46e7-8020-de86111d7478\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " Jul 21 09:05:54.361450 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361350 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-dshm\") pod \"ce2e163a-bf72-46e7-8020-de86111d7478\" (UID: \"ce2e163a-bf72-46e7-8020-de86111d7478\") " Jul 21 09:05:54.361450 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361346 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-home" (OuterVolumeSpecName: "home") pod "ce2e163a-bf72-46e7-8020-de86111d7478" (UID: "ce2e163a-bf72-46e7-8020-de86111d7478"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:05:54.361450 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361360 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce2e163a-bf72-46e7-8020-de86111d7478" (UID: "ce2e163a-bf72-46e7-8020-de86111d7478"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:05:54.361648 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361451 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce2e163a-bf72-46e7-8020-de86111d7478" (UID: "ce2e163a-bf72-46e7-8020-de86111d7478"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:05:54.361648 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361583 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:54.361648 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361596 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:54.361648 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.361608 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:54.363643 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.363617 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-dshm" (OuterVolumeSpecName: "dshm") pod "ce2e163a-bf72-46e7-8020-de86111d7478" (UID: "ce2e163a-bf72-46e7-8020-de86111d7478"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:05:54.363752 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.363699 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2e163a-bf72-46e7-8020-de86111d7478-kube-api-access-nf4hb" (OuterVolumeSpecName: "kube-api-access-nf4hb") pod "ce2e163a-bf72-46e7-8020-de86111d7478" (UID: "ce2e163a-bf72-46e7-8020-de86111d7478"). InnerVolumeSpecName "kube-api-access-nf4hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:05:54.363752 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.363727 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2e163a-bf72-46e7-8020-de86111d7478-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce2e163a-bf72-46e7-8020-de86111d7478" (UID: "ce2e163a-bf72-46e7-8020-de86111d7478"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:05:54.430481 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.430404 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:05:54.462191 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.462161 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf4hb\" (UniqueName: \"kubernetes.io/projected/ce2e163a-bf72-46e7-8020-de86111d7478-kube-api-access-nf4hb\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:54.462191 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.462184 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2e163a-bf72-46e7-8020-de86111d7478-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:54.462191 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.462195 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2e163a-bf72-46e7-8020-de86111d7478-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:05:54.897322 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.897287 2563 generic.go:356] "Generic (PLEG): container finished" podID="ce2e163a-bf72-46e7-8020-de86111d7478" containerID="bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606" exitCode=0 Jul 21 09:05:54.897535 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.897331 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" event={"ID":"ce2e163a-bf72-46e7-8020-de86111d7478","Type":"ContainerDied","Data":"bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606"} Jul 21 09:05:54.897535 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.897335 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" Jul 21 09:05:54.897535 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.897357 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" event={"ID":"ce2e163a-bf72-46e7-8020-de86111d7478","Type":"ContainerDied","Data":"35f228175e6944af40da6926a721e8c448573c0d4b7e0574c00b6007a3a7c47d"} Jul 21 09:05:54.897535 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.897372 2563 scope.go:117] "RemoveContainer" containerID="bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606" Jul 21 09:05:54.906875 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.906857 2563 scope.go:117] "RemoveContainer" containerID="bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606" Jul 21 09:05:54.907116 ip-10-0-137-36 kubenswrapper[2563]: E0721 09:05:54.907099 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606\": container with ID starting with bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606 not found: ID does not exist" containerID="bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606" Jul 21 09:05:54.907169 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.907123 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606"} err="failed to get container status \"bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606\": rpc error: code = NotFound desc = could not find container \"bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606\": container with ID starting with bea5d1636bb1bd932e926dbcded300e914ac2b02f3085de8a23981e219061606 not found: ID does not exist" Jul 21 09:05:54.919679 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.919655 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq"] Jul 21 09:05:54.923996 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:54.923973 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq"] Jul 21 09:05:56.358321 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.358283 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2e163a-bf72-46e7-8020-de86111d7478" path="/var/lib/kubelet/pods/ce2e163a-bf72-46e7-8020-de86111d7478/volumes" Jul 21 09:05:56.852813 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.852784 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29"] Jul 21 09:05:56.853116 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.853105 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b10b5ab7-a357-445f-ba3c-32e716496886" containerName="main" Jul 21 09:05:56.853160 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.853118 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b10b5ab7-a357-445f-ba3c-32e716496886" containerName="main" Jul 21 09:05:56.853160 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.853133 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2e163a-bf72-46e7-8020-de86111d7478" containerName="main" Jul 21 09:05:56.853160 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.853138 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2e163a-bf72-46e7-8020-de86111d7478" containerName="main" Jul 21 09:05:56.853256 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.853195 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2e163a-bf72-46e7-8020-de86111d7478" containerName="main" Jul 21 09:05:56.853256 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.853203 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b10b5ab7-a357-445f-ba3c-32e716496886" containerName="main" Jul 21 09:05:56.856146 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.856129 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:56.862845 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.862825 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-824408\"/\"kube-root-ca.crt\"" Jul 21 09:05:56.862956 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.862828 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-824408\"/\"default-dockercfg-n8xwq\"" Jul 21 09:05:56.862956 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.862881 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-824408\"/\"sub-test-model-824408-kserve-self-signed-certs\"" Jul 21 09:05:56.862956 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.862910 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-824408\"/\"openshift-service-ca.crt\"" Jul 21 09:05:56.868528 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.868507 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29"] Jul 21 09:05:56.880770 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.880743 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:05:56.984806 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.984771 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9dt\" (UniqueName: \"kubernetes.io/projected/219f1394-bf24-40c6-ba83-07e5e0dd163f-kube-api-access-hd9dt\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:56.984982 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.984817 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-tmp-dir\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:56.984982 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.984893 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-home\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:56.984982 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.984926 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219f1394-bf24-40c6-ba83-07e5e0dd163f-tls-certs\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:56.984982 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.984949 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-model-cache\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:56.984982 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:56.984972 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-dshm\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.085733 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.085695 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-home\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.085733 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.085743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219f1394-bf24-40c6-ba83-07e5e0dd163f-tls-certs\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.085975 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.085764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-model-cache\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.085975 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.085786 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-dshm\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.085975 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.085815 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9dt\" (UniqueName: \"kubernetes.io/projected/219f1394-bf24-40c6-ba83-07e5e0dd163f-kube-api-access-hd9dt\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.085975 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.085843 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-tmp-dir\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.086216 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.086190 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-model-cache\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.086281 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.086263 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-tmp-dir\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.086281 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.086267 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-home\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.088307 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.088276 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-dshm\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.088427 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.088403 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219f1394-bf24-40c6-ba83-07e5e0dd163f-tls-certs\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.093943 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.093924 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9dt\" (UniqueName: \"kubernetes.io/projected/219f1394-bf24-40c6-ba83-07e5e0dd163f-kube-api-access-hd9dt\") pod \"sub-test-model-824408-kserve-568f59595-5pg29\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.165087 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.165059 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:05:57.302118 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.302089 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29"] Jul 21 09:05:57.305690 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:05:57.305663 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod219f1394_bf24_40c6_ba83_07e5e0dd163f.slice/crio-c2fce1862e72f12cb2aacf4ae6738adf334f3476986aa898dc2a90126f811b7a WatchSource:0}: Error finding container c2fce1862e72f12cb2aacf4ae6738adf334f3476986aa898dc2a90126f811b7a: Status 404 returned error can't find the container with id c2fce1862e72f12cb2aacf4ae6738adf334f3476986aa898dc2a90126f811b7a Jul 21 09:05:57.918286 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.918256 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" event={"ID":"219f1394-bf24-40c6-ba83-07e5e0dd163f","Type":"ContainerStarted","Data":"2f5a8609bcacd5607da8b1f3cf6a882ec171e9a0dbe75b6a35838eaf856d155f"} Jul 21 09:05:57.918697 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.918295 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" event={"ID":"219f1394-bf24-40c6-ba83-07e5e0dd163f","Type":"ContainerStarted","Data":"c2fce1862e72f12cb2aacf4ae6738adf334f3476986aa898dc2a90126f811b7a"} Jul 21 09:05:57.938938 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:57.938890 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" podStartSLOduration=1.938875219 podStartE2EDuration="1.938875219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:05:57.936227573 +0000 UTC m=+2830.015265905" watchObservedRunningTime="2026-07-21 09:05:57.938875219 +0000 UTC m=+2830.017913551" Jul 21 09:05:59.156790 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:05:59.156744 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-26961c/auth-test-model-26961c-kserve-849d8bcf67-6j5cq" podUID="ce2e163a-bf72-46e7-8020-de86111d7478" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Jul 21 09:06:01.936010 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:01.935977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:06:07.165787 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:07.165750 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:06:07.166294 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:07.165810 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:06:07.178271 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:07.178243 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:06:07.981036 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:07.981002 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:06:15.933565 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:15.933528 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599"] Jul 21 09:06:15.937004 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:15.936986 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:15.952652 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:15.952631 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"kube-root-ca.crt\"" Jul 21 09:06:15.953646 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:15.953624 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"openshift-service-ca.crt\"" Jul 21 09:06:15.953767 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:15.953630 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"default-dockercfg-xnp7p\"" Jul 21 09:06:15.953767 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:15.953744 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"sub-test-model-26961c-kserve-self-signed-certs\"" Jul 21 09:06:15.974064 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:15.974042 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599"] Jul 21 09:06:15.992622 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:15.992593 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:06:16.056269 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.056242 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48qq\" (UniqueName: \"kubernetes.io/projected/601cd96a-29a4-41a4-a407-7de163eb5d9d-kube-api-access-t48qq\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.056461 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.056279 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-model-cache\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.056461 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.056325 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-home\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.056461 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.056364 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-tmp-dir\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.056461 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.056402 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601cd96a-29a4-41a4-a407-7de163eb5d9d-tls-certs\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.056614 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.056469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-dshm\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.157763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.157736 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-home\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.157954 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.157786 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-tmp-dir\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.157954 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.157801 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601cd96a-29a4-41a4-a407-7de163eb5d9d-tls-certs\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.157954 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.157826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-dshm\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.158136 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.157987 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48qq\" (UniqueName: \"kubernetes.io/projected/601cd96a-29a4-41a4-a407-7de163eb5d9d-kube-api-access-t48qq\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.158136 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.158023 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-model-cache\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.158136 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.158123 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-home\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.158267 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.158188 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-tmp-dir\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.158331 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.158314 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-model-cache\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.160183 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.160163 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-dshm\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.160527 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.160512 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601cd96a-29a4-41a4-a407-7de163eb5d9d-tls-certs\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.167503 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.167481 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48qq\" (UniqueName: \"kubernetes.io/projected/601cd96a-29a4-41a4-a407-7de163eb5d9d-kube-api-access-t48qq\") pod \"sub-test-model-26961c-kserve-57d84b8f69-54599\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.246207 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.246105 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:16.386501 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:16.386469 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599"] Jul 21 09:06:16.388748 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:06:16.388725 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod601cd96a_29a4_41a4_a407_7de163eb5d9d.slice/crio-d3200f7c4d61f7962a89fa5a65db6b73f4d0b38bc213c70216b3952d7d499729 WatchSource:0}: Error finding container d3200f7c4d61f7962a89fa5a65db6b73f4d0b38bc213c70216b3952d7d499729: Status 404 returned error can't find the container with id d3200f7c4d61f7962a89fa5a65db6b73f4d0b38bc213c70216b3952d7d499729 Jul 21 09:06:17.017543 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:17.017507 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" event={"ID":"601cd96a-29a4-41a4-a407-7de163eb5d9d","Type":"ContainerStarted","Data":"590ddf7b1f020bd2ed8eef8e51aa909eb9596f0ae78802fa209f3679c10e2c53"} Jul 21 09:06:17.017934 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:17.017552 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" event={"ID":"601cd96a-29a4-41a4-a407-7de163eb5d9d","Type":"ContainerStarted","Data":"d3200f7c4d61f7962a89fa5a65db6b73f4d0b38bc213c70216b3952d7d499729"} Jul 21 09:06:17.048402 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:17.048338 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" podStartSLOduration=2.048317359 podStartE2EDuration="2.048317359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:06:17.046466566 +0000 UTC m=+2849.125504896" watchObservedRunningTime="2026-07-21 09:06:17.048317359 +0000 UTC m=+2849.127355691" Jul 21 09:06:20.048046 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:20.048005 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:06:26.246470 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:26.246436 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:26.246899 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:26.246495 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:26.259366 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:26.259345 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:27.082951 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:27.082922 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:06:36.165398 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:36.165355 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:06:56.264776 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:06:56.264744 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:07:01.197228 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:01.197192 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29"] Jul 21 09:07:01.224507 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:01.224474 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:07:02.077978 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:02.077949 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599"] Jul 21 09:07:03.316222 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:03.316189 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:07:10.906324 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:10.906289 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9"] Jul 21 09:07:10.909777 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:10.909757 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:10.912799 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:10.912778 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-824408\"/\"rate-test-model-824408-kserve-self-signed-certs\"" Jul 21 09:07:10.926634 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:10.926614 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9"] Jul 21 09:07:10.947785 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:10.947755 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:07:11.026854 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.026822 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-model-cache\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.027034 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.026859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-tmp-dir\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.027034 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.026903 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-home\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.027034 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.026951 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85xm6\" (UniqueName: \"kubernetes.io/projected/133415d6-438a-4ef7-94cd-b7b62da70014-kube-api-access-85xm6\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.027034 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.026998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-dshm\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.027227 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.027084 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133415d6-438a-4ef7-94cd-b7b62da70014-tls-certs\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.128182 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.128141 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-home\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.128442 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.128230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85xm6\" (UniqueName: \"kubernetes.io/projected/133415d6-438a-4ef7-94cd-b7b62da70014-kube-api-access-85xm6\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.128442 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.128311 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-dshm\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.128442 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.128349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133415d6-438a-4ef7-94cd-b7b62da70014-tls-certs\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.128442 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.128422 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-model-cache\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.128662 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.128452 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-tmp-dir\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.128662 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.128612 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-home\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.128835 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.128815 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-model-cache\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.128894 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.128848 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-tmp-dir\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.130850 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.130826 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-dshm\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.131051 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.131033 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133415d6-438a-4ef7-94cd-b7b62da70014-tls-certs\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.136894 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.136872 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85xm6\" (UniqueName: \"kubernetes.io/projected/133415d6-438a-4ef7-94cd-b7b62da70014-kube-api-access-85xm6\") pod \"rate-test-model-824408-kserve-5cdbdc84b7-wg9p9\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.219338 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.219254 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:11.361116 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:11.361088 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9"] Jul 21 09:07:11.363033 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:07:11.363005 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod133415d6_438a_4ef7_94cd_b7b62da70014.slice/crio-49f0a66c16898010d927d74c66c0cc7a1e842a7d35c0aff8e57490d523a2008b WatchSource:0}: Error finding container 49f0a66c16898010d927d74c66c0cc7a1e842a7d35c0aff8e57490d523a2008b: Status 404 returned error can't find the container with id 49f0a66c16898010d927d74c66c0cc7a1e842a7d35c0aff8e57490d523a2008b Jul 21 09:07:12.309279 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:12.309242 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" event={"ID":"133415d6-438a-4ef7-94cd-b7b62da70014","Type":"ContainerStarted","Data":"970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821"} Jul 21 09:07:12.309279 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:12.309292 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" event={"ID":"133415d6-438a-4ef7-94cd-b7b62da70014","Type":"ContainerStarted","Data":"49f0a66c16898010d927d74c66c0cc7a1e842a7d35c0aff8e57490d523a2008b"} Jul 21 09:07:12.327727 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:12.327679 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" podStartSLOduration=2.32766406 podStartE2EDuration="2.32766406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:07:12.325968419 +0000 UTC m=+2904.405006753" watchObservedRunningTime="2026-07-21 09:07:12.32766406 +0000 UTC m=+2904.406702390" Jul 21 09:07:13.329285 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:13.329255 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:07:16.217648 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.217609 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" podUID="219f1394-bf24-40c6-ba83-07e5e0dd163f" containerName="main" containerID="cri-o://2f5a8609bcacd5607da8b1f3cf6a882ec171e9a0dbe75b6a35838eaf856d155f" gracePeriod=45 Jul 21 09:07:16.333608 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.333578 2563 generic.go:356] "Generic (PLEG): container finished" podID="219f1394-bf24-40c6-ba83-07e5e0dd163f" containerID="2f5a8609bcacd5607da8b1f3cf6a882ec171e9a0dbe75b6a35838eaf856d155f" exitCode=0 Jul 21 09:07:16.333793 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.333614 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" event={"ID":"219f1394-bf24-40c6-ba83-07e5e0dd163f","Type":"ContainerDied","Data":"2f5a8609bcacd5607da8b1f3cf6a882ec171e9a0dbe75b6a35838eaf856d155f"} Jul 21 09:07:16.473087 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.473027 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:07:16.574481 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574454 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-dshm\") pod \"219f1394-bf24-40c6-ba83-07e5e0dd163f\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " Jul 21 09:07:16.574673 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574493 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-tmp-dir\") pod \"219f1394-bf24-40c6-ba83-07e5e0dd163f\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " Jul 21 09:07:16.574673 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574510 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-home\") pod \"219f1394-bf24-40c6-ba83-07e5e0dd163f\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " Jul 21 09:07:16.574673 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574525 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-model-cache\") pod \"219f1394-bf24-40c6-ba83-07e5e0dd163f\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " Jul 21 09:07:16.574673 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574573 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd9dt\" (UniqueName: \"kubernetes.io/projected/219f1394-bf24-40c6-ba83-07e5e0dd163f-kube-api-access-hd9dt\") pod \"219f1394-bf24-40c6-ba83-07e5e0dd163f\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " Jul 21 09:07:16.574673 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574609 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219f1394-bf24-40c6-ba83-07e5e0dd163f-tls-certs\") pod \"219f1394-bf24-40c6-ba83-07e5e0dd163f\" (UID: \"219f1394-bf24-40c6-ba83-07e5e0dd163f\") " Jul 21 09:07:16.574917 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574818 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-home" (OuterVolumeSpecName: "home") pod "219f1394-bf24-40c6-ba83-07e5e0dd163f" (UID: "219f1394-bf24-40c6-ba83-07e5e0dd163f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:07:16.574917 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574823 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "219f1394-bf24-40c6-ba83-07e5e0dd163f" (UID: "219f1394-bf24-40c6-ba83-07e5e0dd163f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:07:16.574917 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574892 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:16.574917 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.574909 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:16.575150 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.575128 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-model-cache" (OuterVolumeSpecName: "model-cache") pod "219f1394-bf24-40c6-ba83-07e5e0dd163f" (UID: "219f1394-bf24-40c6-ba83-07e5e0dd163f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:07:16.576854 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.576823 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-dshm" (OuterVolumeSpecName: "dshm") pod "219f1394-bf24-40c6-ba83-07e5e0dd163f" (UID: "219f1394-bf24-40c6-ba83-07e5e0dd163f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:07:16.577558 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.577534 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219f1394-bf24-40c6-ba83-07e5e0dd163f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "219f1394-bf24-40c6-ba83-07e5e0dd163f" (UID: "219f1394-bf24-40c6-ba83-07e5e0dd163f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:07:16.577654 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.577537 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219f1394-bf24-40c6-ba83-07e5e0dd163f-kube-api-access-hd9dt" (OuterVolumeSpecName: "kube-api-access-hd9dt") pod "219f1394-bf24-40c6-ba83-07e5e0dd163f" (UID: "219f1394-bf24-40c6-ba83-07e5e0dd163f"). InnerVolumeSpecName "kube-api-access-hd9dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:07:16.675325 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.675293 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:16.675325 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.675319 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f1394-bf24-40c6-ba83-07e5e0dd163f-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:16.675325 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.675329 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd9dt\" (UniqueName: \"kubernetes.io/projected/219f1394-bf24-40c6-ba83-07e5e0dd163f-kube-api-access-hd9dt\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:16.675588 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:16.675338 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219f1394-bf24-40c6-ba83-07e5e0dd163f-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:17.098939 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.098894 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" podUID="601cd96a-29a4-41a4-a407-7de163eb5d9d" containerName="main" containerID="cri-o://590ddf7b1f020bd2ed8eef8e51aa909eb9596f0ae78802fa209f3679c10e2c53" gracePeriod=45 Jul 21 09:07:17.340650 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.340609 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" Jul 21 09:07:17.340940 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.340648 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29" event={"ID":"219f1394-bf24-40c6-ba83-07e5e0dd163f","Type":"ContainerDied","Data":"c2fce1862e72f12cb2aacf4ae6738adf334f3476986aa898dc2a90126f811b7a"} Jul 21 09:07:17.340940 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.340701 2563 scope.go:117] "RemoveContainer" containerID="2f5a8609bcacd5607da8b1f3cf6a882ec171e9a0dbe75b6a35838eaf856d155f" Jul 21 09:07:17.342332 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.342311 2563 generic.go:356] "Generic (PLEG): container finished" podID="601cd96a-29a4-41a4-a407-7de163eb5d9d" containerID="590ddf7b1f020bd2ed8eef8e51aa909eb9596f0ae78802fa209f3679c10e2c53" exitCode=0 Jul 21 09:07:17.342459 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.342402 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" event={"ID":"601cd96a-29a4-41a4-a407-7de163eb5d9d","Type":"ContainerDied","Data":"590ddf7b1f020bd2ed8eef8e51aa909eb9596f0ae78802fa209f3679c10e2c53"} Jul 21 09:07:17.367051 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.367033 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:07:17.379842 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.379817 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29"] Jul 21 09:07:17.380636 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.380618 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t48qq\" (UniqueName: \"kubernetes.io/projected/601cd96a-29a4-41a4-a407-7de163eb5d9d-kube-api-access-t48qq\") pod \"601cd96a-29a4-41a4-a407-7de163eb5d9d\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " Jul 21 09:07:17.380739 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.380703 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-dshm\") pod \"601cd96a-29a4-41a4-a407-7de163eb5d9d\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " Jul 21 09:07:17.380739 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.380734 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601cd96a-29a4-41a4-a407-7de163eb5d9d-tls-certs\") pod \"601cd96a-29a4-41a4-a407-7de163eb5d9d\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " Jul 21 09:07:17.380854 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.380788 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-model-cache\") pod \"601cd96a-29a4-41a4-a407-7de163eb5d9d\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " Jul 21 09:07:17.380854 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.380833 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-tmp-dir\") pod \"601cd96a-29a4-41a4-a407-7de163eb5d9d\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " Jul 21 09:07:17.380955 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.380877 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-home\") pod \"601cd96a-29a4-41a4-a407-7de163eb5d9d\" (UID: \"601cd96a-29a4-41a4-a407-7de163eb5d9d\") " Jul 21 09:07:17.381082 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.381059 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "601cd96a-29a4-41a4-a407-7de163eb5d9d" (UID: "601cd96a-29a4-41a4-a407-7de163eb5d9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:07:17.381160 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.381139 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "601cd96a-29a4-41a4-a407-7de163eb5d9d" (UID: "601cd96a-29a4-41a4-a407-7de163eb5d9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:07:17.381261 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.381242 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-home" (OuterVolumeSpecName: "home") pod "601cd96a-29a4-41a4-a407-7de163eb5d9d" (UID: "601cd96a-29a4-41a4-a407-7de163eb5d9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:07:17.381711 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.381492 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:17.381711 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.381683 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:17.381711 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.381703 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:17.383512 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.383483 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-dshm" (OuterVolumeSpecName: "dshm") pod "601cd96a-29a4-41a4-a407-7de163eb5d9d" (UID: "601cd96a-29a4-41a4-a407-7de163eb5d9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:07:17.383752 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.383732 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601cd96a-29a4-41a4-a407-7de163eb5d9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "601cd96a-29a4-41a4-a407-7de163eb5d9d" (UID: "601cd96a-29a4-41a4-a407-7de163eb5d9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:07:17.383871 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.383854 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601cd96a-29a4-41a4-a407-7de163eb5d9d-kube-api-access-t48qq" (OuterVolumeSpecName: "kube-api-access-t48qq") pod "601cd96a-29a4-41a4-a407-7de163eb5d9d" (UID: "601cd96a-29a4-41a4-a407-7de163eb5d9d"). InnerVolumeSpecName "kube-api-access-t48qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:07:17.384968 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.384949 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-824408/sub-test-model-824408-kserve-568f59595-5pg29"] Jul 21 09:07:17.482728 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.482698 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t48qq\" (UniqueName: \"kubernetes.io/projected/601cd96a-29a4-41a4-a407-7de163eb5d9d-kube-api-access-t48qq\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:17.482728 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.482726 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601cd96a-29a4-41a4-a407-7de163eb5d9d-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:17.482937 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:17.482737 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601cd96a-29a4-41a4-a407-7de163eb5d9d-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:07:18.348886 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:18.348855 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" Jul 21 09:07:18.349303 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:18.348896 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599" event={"ID":"601cd96a-29a4-41a4-a407-7de163eb5d9d","Type":"ContainerDied","Data":"d3200f7c4d61f7962a89fa5a65db6b73f4d0b38bc213c70216b3952d7d499729"} Jul 21 09:07:18.349303 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:18.348948 2563 scope.go:117] "RemoveContainer" containerID="590ddf7b1f020bd2ed8eef8e51aa909eb9596f0ae78802fa209f3679c10e2c53" Jul 21 09:07:18.358778 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:18.358758 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="219f1394-bf24-40c6-ba83-07e5e0dd163f" path="/var/lib/kubelet/pods/219f1394-bf24-40c6-ba83-07e5e0dd163f/volumes" Jul 21 09:07:18.371329 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:18.371300 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599"] Jul 21 09:07:18.377261 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:18.377241 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/sub-test-model-26961c-kserve-57d84b8f69-54599"] Jul 21 09:07:20.358314 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:20.358279 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="601cd96a-29a4-41a4-a407-7de163eb5d9d" path="/var/lib/kubelet/pods/601cd96a-29a4-41a4-a407-7de163eb5d9d/volumes" Jul 21 09:07:21.220234 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:21.220183 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:21.220460 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:21.220249 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:21.233181 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:21.233153 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:21.376300 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:21.376266 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:07:39.319652 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:39.319613 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:07:39.419341 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:39.419310 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:07:51.395028 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:51.394989 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:07:51.494417 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:51.494367 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:07:56.822505 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.822472 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc"] Jul 21 09:07:56.822856 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.822821 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219f1394-bf24-40c6-ba83-07e5e0dd163f" containerName="main" Jul 21 09:07:56.822856 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.822832 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="219f1394-bf24-40c6-ba83-07e5e0dd163f" containerName="main" Jul 21 09:07:56.822856 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.822850 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601cd96a-29a4-41a4-a407-7de163eb5d9d" containerName="main" Jul 21 09:07:56.822856 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.822856 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="601cd96a-29a4-41a4-a407-7de163eb5d9d" containerName="main" Jul 21 09:07:56.823035 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.822906 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="601cd96a-29a4-41a4-a407-7de163eb5d9d" containerName="main" Jul 21 09:07:56.823035 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.822915 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="219f1394-bf24-40c6-ba83-07e5e0dd163f" containerName="main" Jul 21 09:07:56.825778 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.825763 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:56.828641 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.828618 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"default-dockercfg-xnp7p\"" Jul 21 09:07:56.829622 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.829605 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"kube-root-ca.crt\"" Jul 21 09:07:56.829727 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.829610 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"rate-test-model-26961c-kserve-self-signed-certs\"" Jul 21 09:07:56.829727 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.829642 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26961c\"/\"openshift-service-ca.crt\"" Jul 21 09:07:56.843905 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.843883 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc"] Jul 21 09:07:56.881783 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.881746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:07:56.934590 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.934557 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-model-cache\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:56.934763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.934614 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tls-certs\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:56.934763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.934695 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-dshm\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:56.934763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.934728 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tmp-dir\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:56.934763 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.934752 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q57hw\" (UniqueName: \"kubernetes.io/projected/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-kube-api-access-q57hw\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:56.934914 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:56.934807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-home\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.035564 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.035525 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q57hw\" (UniqueName: \"kubernetes.io/projected/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-kube-api-access-q57hw\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.035564 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.035567 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-home\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.035804 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.035606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-model-cache\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.035804 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.035646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tls-certs\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.035804 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.035681 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-dshm\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.035804 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.035712 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tmp-dir\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.036135 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.036113 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-model-cache\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.036201 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.036108 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-home\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.036201 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.036172 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tmp-dir\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.038138 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.038118 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-dshm\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.038311 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.038295 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tls-certs\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.046089 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.046048 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q57hw\" (UniqueName: \"kubernetes.io/projected/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-kube-api-access-q57hw\") pod \"rate-test-model-26961c-kserve-766594c859-v49sc\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.134924 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.134834 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:07:57.269770 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.269746 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc"] Jul 21 09:07:57.271917 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:07:57.271890 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba4b2aa4_7dba_4d85_8b0b_cb4db2784820.slice/crio-e2e838317de91b963a5c8493116a79a8b086428e9ad76751839b63bdf6b17c0c WatchSource:0}: Error finding container e2e838317de91b963a5c8493116a79a8b086428e9ad76751839b63bdf6b17c0c: Status 404 returned error can't find the container with id e2e838317de91b963a5c8493116a79a8b086428e9ad76751839b63bdf6b17c0c Jul 21 09:07:57.543848 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.543813 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" event={"ID":"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820","Type":"ContainerStarted","Data":"45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511"} Jul 21 09:07:57.543848 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.543856 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" event={"ID":"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820","Type":"ContainerStarted","Data":"e2e838317de91b963a5c8493116a79a8b086428e9ad76751839b63bdf6b17c0c"} Jul 21 09:07:57.567667 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:07:57.567616 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" podStartSLOduration=1.5676014029999998 podStartE2EDuration="1.567601403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:07:57.566558629 +0000 UTC m=+2949.645596961" watchObservedRunningTime="2026-07-21 09:07:57.567601403 +0000 UTC m=+2949.646639734" Jul 21 09:08:00.232657 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:00.232618 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:08:01.020977 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:01.020940 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:08:01.121724 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:01.121691 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:08:07.135078 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:07.135033 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:08:07.135507 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:07.135113 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:08:07.147747 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:07.147722 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:08:07.604734 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:07.604705 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:08:14.741860 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:14.741782 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:08:24.239100 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:24.239030 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:08:24.384619 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:24.384583 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:08:29.937815 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:29.937774 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:08:37.323589 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:37.323558 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9"] Jul 21 09:08:37.356139 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:37.356106 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:08:38.142646 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:38.142609 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:08:38.386756 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:38.386724 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc"] Jul 21 09:08:38.393936 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:38.393851 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:08:39.437790 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:39.437753 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-140180322520496"] Jul 21 09:08:39.441465 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:39.441441 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322520496" Jul 21 09:08:39.444133 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:39.444109 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4dq68\"" Jul 21 09:08:39.450509 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:39.450490 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322520496"] Jul 21 09:08:39.616543 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:39.616512 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvms\" (UniqueName: \"kubernetes.io/projected/b8ee784d-a8fd-4af9-a660-9f74a8029b89-kube-api-access-qnvms\") pod \"test-curl-7481-140180322520496\" (UID: \"b8ee784d-a8fd-4af9-a660-9f74a8029b89\") " pod="opendatahub/test-curl-7481-140180322520496" Jul 21 09:08:39.717946 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:39.717855 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvms\" (UniqueName: \"kubernetes.io/projected/b8ee784d-a8fd-4af9-a660-9f74a8029b89-kube-api-access-qnvms\") pod \"test-curl-7481-140180322520496\" (UID: \"b8ee784d-a8fd-4af9-a660-9f74a8029b89\") " pod="opendatahub/test-curl-7481-140180322520496" Jul 21 09:08:39.726734 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:39.726710 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvms\" (UniqueName: \"kubernetes.io/projected/b8ee784d-a8fd-4af9-a660-9f74a8029b89-kube-api-access-qnvms\") pod \"test-curl-7481-140180322520496\" (UID: \"b8ee784d-a8fd-4af9-a660-9f74a8029b89\") " pod="opendatahub/test-curl-7481-140180322520496" Jul 21 09:08:39.751625 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:39.751601 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322520496" Jul 21 09:08:40.103206 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:40.103181 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322520496"] Jul 21 09:08:40.105024 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:08:40.104995 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8ee784d_a8fd_4af9_a660_9f74a8029b89.slice/crio-ea81013706fe803e0c8eab0377da4d606164b849297a219d177538a62053951d WatchSource:0}: Error finding container ea81013706fe803e0c8eab0377da4d606164b849297a219d177538a62053951d: Status 404 returned error can't find the container with id ea81013706fe803e0c8eab0377da4d606164b849297a219d177538a62053951d Jul 21 09:08:40.106349 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:40.106332 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 09:08:40.773888 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:40.773855 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322520496" event={"ID":"b8ee784d-a8fd-4af9-a660-9f74a8029b89","Type":"ContainerStarted","Data":"ea81013706fe803e0c8eab0377da4d606164b849297a219d177538a62053951d"} Jul 21 09:08:41.780466 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:41.780430 2563 generic.go:356] "Generic (PLEG): container finished" podID="b8ee784d-a8fd-4af9-a660-9f74a8029b89" containerID="c0b4fdc39e5778e3cc4cd5bd9685f445d141cb7242ffaf726b106c793fe45134" exitCode=0 Jul 21 09:08:41.780833 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:41.780512 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322520496" event={"ID":"b8ee784d-a8fd-4af9-a660-9f74a8029b89","Type":"ContainerDied","Data":"c0b4fdc39e5778e3cc4cd5bd9685f445d141cb7242ffaf726b106c793fe45134"} Jul 21 09:08:42.922740 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:42.922713 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322520496" Jul 21 09:08:42.947711 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:42.947665 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-140180322520496_b8ee784d-a8fd-4af9-a660-9f74a8029b89/test-curl-7481-140180322520496/0.log" Jul 21 09:08:42.974771 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:42.974745 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-140180322520496"] Jul 21 09:08:42.976623 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:42.976604 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-140180322520496"] Jul 21 09:08:43.049076 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.048996 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnvms\" (UniqueName: \"kubernetes.io/projected/b8ee784d-a8fd-4af9-a660-9f74a8029b89-kube-api-access-qnvms\") pod \"b8ee784d-a8fd-4af9-a660-9f74a8029b89\" (UID: \"b8ee784d-a8fd-4af9-a660-9f74a8029b89\") " Jul 21 09:08:43.051300 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.051272 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ee784d-a8fd-4af9-a660-9f74a8029b89-kube-api-access-qnvms" (OuterVolumeSpecName: "kube-api-access-qnvms") pod "b8ee784d-a8fd-4af9-a660-9f74a8029b89" (UID: "b8ee784d-a8fd-4af9-a660-9f74a8029b89"). InnerVolumeSpecName "kube-api-access-qnvms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:08:43.109103 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.109065 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-140180322443824"] Jul 21 09:08:43.109493 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.109476 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ee784d-a8fd-4af9-a660-9f74a8029b89" containerName="test-curl-7481-140180322520496" Jul 21 09:08:43.109589 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.109495 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ee784d-a8fd-4af9-a660-9f74a8029b89" containerName="test-curl-7481-140180322520496" Jul 21 09:08:43.109644 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.109610 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8ee784d-a8fd-4af9-a660-9f74a8029b89" containerName="test-curl-7481-140180322520496" Jul 21 09:08:43.112565 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.112547 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322443824" Jul 21 09:08:43.120003 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.119976 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322443824"] Jul 21 09:08:43.150743 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.150719 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnvms\" (UniqueName: \"kubernetes.io/projected/b8ee784d-a8fd-4af9-a660-9f74a8029b89-kube-api-access-qnvms\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:43.251170 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.251137 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6t8\" (UniqueName: \"kubernetes.io/projected/c94d01b4-831e-4800-b777-1de1995f23c7-kube-api-access-kh6t8\") pod \"test-curl-7481-140180322443824\" (UID: \"c94d01b4-831e-4800-b777-1de1995f23c7\") " pod="opendatahub/test-curl-7481-140180322443824" Jul 21 09:08:43.352107 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.352012 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6t8\" (UniqueName: \"kubernetes.io/projected/c94d01b4-831e-4800-b777-1de1995f23c7-kube-api-access-kh6t8\") pod \"test-curl-7481-140180322443824\" (UID: \"c94d01b4-831e-4800-b777-1de1995f23c7\") " pod="opendatahub/test-curl-7481-140180322443824" Jul 21 09:08:43.361974 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.361952 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6t8\" (UniqueName: \"kubernetes.io/projected/c94d01b4-831e-4800-b777-1de1995f23c7-kube-api-access-kh6t8\") pod \"test-curl-7481-140180322443824\" (UID: \"c94d01b4-831e-4800-b777-1de1995f23c7\") " pod="opendatahub/test-curl-7481-140180322443824" Jul 21 09:08:43.422759 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.422729 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322443824" Jul 21 09:08:43.557293 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.557270 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322443824"] Jul 21 09:08:43.559089 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:08:43.559063 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc94d01b4_831e_4800_b777_1de1995f23c7.slice/crio-e6c1e517a6e2f41bd8a1c04356459e76926194ade2ef918662a089ed86114a23 WatchSource:0}: Error finding container e6c1e517a6e2f41bd8a1c04356459e76926194ade2ef918662a089ed86114a23: Status 404 returned error can't find the container with id e6c1e517a6e2f41bd8a1c04356459e76926194ade2ef918662a089ed86114a23 Jul 21 09:08:43.791741 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.791696 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322443824" event={"ID":"c94d01b4-831e-4800-b777-1de1995f23c7","Type":"ContainerStarted","Data":"e6c1e517a6e2f41bd8a1c04356459e76926194ade2ef918662a089ed86114a23"} Jul 21 09:08:43.792969 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.792947 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322520496" Jul 21 09:08:43.793107 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:43.792951 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea81013706fe803e0c8eab0377da4d606164b849297a219d177538a62053951d" Jul 21 09:08:43.803610 ip-10-0-137-36 kubenswrapper[2563]: E0721 09:08:43.803583 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7481-140180322520496\" is forbidden: User \"system:node:ip-10-0-137-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-36.ec2.internal' and this object" podUID="b8ee784d-a8fd-4af9-a660-9f74a8029b89" pod="opendatahub/test-curl-7481-140180322520496" Jul 21 09:08:44.359955 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:44.359921 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8ee784d-a8fd-4af9-a660-9f74a8029b89" path="/var/lib/kubelet/pods/b8ee784d-a8fd-4af9-a660-9f74a8029b89/volumes" Jul 21 09:08:44.567744 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:44.567704 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:08:44.799053 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:44.799008 2563 generic.go:356] "Generic (PLEG): container finished" podID="c94d01b4-831e-4800-b777-1de1995f23c7" containerID="e4729abd57c4bf70dfb5fe643dd462c6ec707ba74e655ac0abb674f71f7e1e31" exitCode=0 Jul 21 09:08:44.799263 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:44.799101 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322443824" event={"ID":"c94d01b4-831e-4800-b777-1de1995f23c7","Type":"ContainerDied","Data":"e4729abd57c4bf70dfb5fe643dd462c6ec707ba74e655ac0abb674f71f7e1e31"} Jul 21 09:08:45.940786 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:45.940760 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322443824" Jul 21 09:08:45.963485 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:45.963458 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-140180322443824_c94d01b4-831e-4800-b777-1de1995f23c7/test-curl-7481-140180322443824/0.log" Jul 21 09:08:45.990801 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:45.990773 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-140180322443824"] Jul 21 09:08:45.996718 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:45.996696 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-140180322443824"] Jul 21 09:08:46.077330 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.077245 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh6t8\" (UniqueName: \"kubernetes.io/projected/c94d01b4-831e-4800-b777-1de1995f23c7-kube-api-access-kh6t8\") pod \"c94d01b4-831e-4800-b777-1de1995f23c7\" (UID: \"c94d01b4-831e-4800-b777-1de1995f23c7\") " Jul 21 09:08:46.079486 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.079466 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94d01b4-831e-4800-b777-1de1995f23c7-kube-api-access-kh6t8" (OuterVolumeSpecName: "kube-api-access-kh6t8") pod "c94d01b4-831e-4800-b777-1de1995f23c7" (UID: "c94d01b4-831e-4800-b777-1de1995f23c7"). InnerVolumeSpecName "kube-api-access-kh6t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:08:46.163816 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.163780 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:46.164134 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.164121 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c94d01b4-831e-4800-b777-1de1995f23c7" containerName="test-curl-7481-140180322443824" Jul 21 09:08:46.164178 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.164135 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c94d01b4-831e-4800-b777-1de1995f23c7" containerName="test-curl-7481-140180322443824" Jul 21 09:08:46.164212 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.164190 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c94d01b4-831e-4800-b777-1de1995f23c7" containerName="test-curl-7481-140180322443824" Jul 21 09:08:46.167105 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.167089 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:46.177001 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.176977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:46.178925 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.178903 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6t8\" (UniqueName: \"kubernetes.io/projected/c94d01b4-831e-4800-b777-1de1995f23c7-kube-api-access-kh6t8\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:46.279998 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.279958 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdl6g\" (UniqueName: \"kubernetes.io/projected/351f871a-c4f5-4203-b271-3cd08b56e620-kube-api-access-qdl6g\") pod \"test-curl-7481-140180322337968\" (UID: \"351f871a-c4f5-4203-b271-3cd08b56e620\") " pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:46.363669 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.358973 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c94d01b4-831e-4800-b777-1de1995f23c7" path="/var/lib/kubelet/pods/c94d01b4-831e-4800-b777-1de1995f23c7/volumes" Jul 21 09:08:46.380564 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.380539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdl6g\" (UniqueName: \"kubernetes.io/projected/351f871a-c4f5-4203-b271-3cd08b56e620-kube-api-access-qdl6g\") pod \"test-curl-7481-140180322337968\" (UID: \"351f871a-c4f5-4203-b271-3cd08b56e620\") " pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:46.392360 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.392332 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdl6g\" (UniqueName: \"kubernetes.io/projected/351f871a-c4f5-4203-b271-3cd08b56e620-kube-api-access-qdl6g\") pod \"test-curl-7481-140180322337968\" (UID: \"351f871a-c4f5-4203-b271-3cd08b56e620\") " pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:46.476527 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.476499 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:46.617744 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.617718 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:46.619826 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:08:46.619799 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod351f871a_c4f5_4203_b271_3cd08b56e620.slice/crio-8b230f201bcf75f3ce9136c901f1a59271a6a67ca62ff31840ce7a939bffa62b WatchSource:0}: Error finding container 8b230f201bcf75f3ce9136c901f1a59271a6a67ca62ff31840ce7a939bffa62b: Status 404 returned error can't find the container with id 8b230f201bcf75f3ce9136c901f1a59271a6a67ca62ff31840ce7a939bffa62b Jul 21 09:08:46.810471 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.810436 2563 scope.go:117] "RemoveContainer" containerID="e4729abd57c4bf70dfb5fe643dd462c6ec707ba74e655ac0abb674f71f7e1e31" Jul 21 09:08:46.810471 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.810460 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322443824" Jul 21 09:08:46.811822 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:46.811799 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322337968" event={"ID":"351f871a-c4f5-4203-b271-3cd08b56e620","Type":"ContainerStarted","Data":"8b230f201bcf75f3ce9136c901f1a59271a6a67ca62ff31840ce7a939bffa62b"} Jul 21 09:08:47.817941 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:47.817901 2563 generic.go:356] "Generic (PLEG): container finished" podID="351f871a-c4f5-4203-b271-3cd08b56e620" containerID="7ed129815658e2d7315521c9461bc19e645e65ed41ef2427e5cebd80d2697c8e" exitCode=0 Jul 21 09:08:47.818439 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:47.817989 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322337968" event={"ID":"351f871a-c4f5-4203-b271-3cd08b56e620","Type":"ContainerDied","Data":"7ed129815658e2d7315521c9461bc19e645e65ed41ef2427e5cebd80d2697c8e"} Jul 21 09:08:48.964964 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:48.964939 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:48.984941 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:48.984915 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-140180322337968_351f871a-c4f5-4203-b271-3cd08b56e620/test-curl-7481-140180322337968/0.log" Jul 21 09:08:49.002487 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.002464 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdl6g\" (UniqueName: \"kubernetes.io/projected/351f871a-c4f5-4203-b271-3cd08b56e620-kube-api-access-qdl6g\") pod \"351f871a-c4f5-4203-b271-3cd08b56e620\" (UID: \"351f871a-c4f5-4203-b271-3cd08b56e620\") " Jul 21 09:08:49.004943 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.004914 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351f871a-c4f5-4203-b271-3cd08b56e620-kube-api-access-qdl6g" (OuterVolumeSpecName: "kube-api-access-qdl6g") pod "351f871a-c4f5-4203-b271-3cd08b56e620" (UID: "351f871a-c4f5-4203-b271-3cd08b56e620"). InnerVolumeSpecName "kube-api-access-qdl6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:08:49.005734 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.005583 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:49.007716 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.007697 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:49.103895 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.103818 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdl6g\" (UniqueName: \"kubernetes.io/projected/351f871a-c4f5-4203-b271-3cd08b56e620-kube-api-access-qdl6g\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:49.142128 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.142102 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:49.142456 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.142442 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351f871a-c4f5-4203-b271-3cd08b56e620" containerName="test-curl-7481-140180322337968" Jul 21 09:08:49.142499 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.142458 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="351f871a-c4f5-4203-b271-3cd08b56e620" containerName="test-curl-7481-140180322337968" Jul 21 09:08:49.142530 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.142515 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="351f871a-c4f5-4203-b271-3cd08b56e620" containerName="test-curl-7481-140180322337968" Jul 21 09:08:49.145431 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.145417 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:49.163592 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.163568 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:49.204359 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.204331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7gz\" (UniqueName: \"kubernetes.io/projected/08105d94-dd52-473a-9045-56a8a7f681f0-kube-api-access-vd7gz\") pod \"test-curl-7481-140180322337968\" (UID: \"08105d94-dd52-473a-9045-56a8a7f681f0\") " pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:49.305590 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.305557 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7gz\" (UniqueName: \"kubernetes.io/projected/08105d94-dd52-473a-9045-56a8a7f681f0-kube-api-access-vd7gz\") pod \"test-curl-7481-140180322337968\" (UID: \"08105d94-dd52-473a-9045-56a8a7f681f0\") " pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:49.315495 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.315473 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7gz\" (UniqueName: \"kubernetes.io/projected/08105d94-dd52-473a-9045-56a8a7f681f0-kube-api-access-vd7gz\") pod \"test-curl-7481-140180322337968\" (UID: \"08105d94-dd52-473a-9045-56a8a7f681f0\") " pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:49.454585 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.454557 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:49.615166 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.615133 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:49.619232 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:08:49.619208 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08105d94_dd52_473a_9045_56a8a7f681f0.slice/crio-f27e56725584fcc358ea2b2d9cdb68895f2bacd3fc37558b9d1560ff48da46bc WatchSource:0}: Error finding container f27e56725584fcc358ea2b2d9cdb68895f2bacd3fc37558b9d1560ff48da46bc: Status 404 returned error can't find the container with id f27e56725584fcc358ea2b2d9cdb68895f2bacd3fc37558b9d1560ff48da46bc Jul 21 09:08:49.832286 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.831258 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b230f201bcf75f3ce9136c901f1a59271a6a67ca62ff31840ce7a939bffa62b" Jul 21 09:08:49.832286 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.831360 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:49.838814 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.838784 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322337968" event={"ID":"08105d94-dd52-473a-9045-56a8a7f681f0","Type":"ContainerStarted","Data":"f27e56725584fcc358ea2b2d9cdb68895f2bacd3fc37558b9d1560ff48da46bc"} Jul 21 09:08:49.846765 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:49.846734 2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7481-140180322337968" oldPodUID="351f871a-c4f5-4203-b271-3cd08b56e620" podUID="08105d94-dd52-473a-9045-56a8a7f681f0" Jul 21 09:08:50.358558 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:50.358524 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351f871a-c4f5-4203-b271-3cd08b56e620" path="/var/lib/kubelet/pods/351f871a-c4f5-4203-b271-3cd08b56e620/volumes" Jul 21 09:08:50.845627 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:50.845588 2563 generic.go:356] "Generic (PLEG): container finished" podID="08105d94-dd52-473a-9045-56a8a7f681f0" containerID="2fb40c462f158ebc628f64e279f852391a88f919a13e8fddff88f5e89457b005" exitCode=0 Jul 21 09:08:50.845802 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:50.845670 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322337968" event={"ID":"08105d94-dd52-473a-9045-56a8a7f681f0","Type":"ContainerDied","Data":"2fb40c462f158ebc628f64e279f852391a88f919a13e8fddff88f5e89457b005"} Jul 21 09:08:51.988509 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:51.988483 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:52.030347 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.030319 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd7gz\" (UniqueName: \"kubernetes.io/projected/08105d94-dd52-473a-9045-56a8a7f681f0-kube-api-access-vd7gz\") pod \"08105d94-dd52-473a-9045-56a8a7f681f0\" (UID: \"08105d94-dd52-473a-9045-56a8a7f681f0\") " Jul 21 09:08:52.032550 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.032520 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08105d94-dd52-473a-9045-56a8a7f681f0-kube-api-access-vd7gz" (OuterVolumeSpecName: "kube-api-access-vd7gz") pod "08105d94-dd52-473a-9045-56a8a7f681f0" (UID: "08105d94-dd52-473a-9045-56a8a7f681f0"). InnerVolumeSpecName "kube-api-access-vd7gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:08:52.035608 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.035582 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-140180322337968_08105d94-dd52-473a-9045-56a8a7f681f0/test-curl-7481-140180322337968/0.log" Jul 21 09:08:52.067299 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.067274 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:52.071912 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.071887 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-140180322337968"] Jul 21 09:08:52.131678 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.131600 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7gz\" (UniqueName: \"kubernetes.io/projected/08105d94-dd52-473a-9045-56a8a7f681f0-kube-api-access-vd7gz\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:52.216025 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.215992 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:52.216332 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.216320 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08105d94-dd52-473a-9045-56a8a7f681f0" containerName="test-curl-7481-140180322337968" Jul 21 09:08:52.216375 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.216334 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="08105d94-dd52-473a-9045-56a8a7f681f0" containerName="test-curl-7481-140180322337968" Jul 21 09:08:52.216431 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.216416 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="08105d94-dd52-473a-9045-56a8a7f681f0" containerName="test-curl-7481-140180322337968" Jul 21 09:08:52.219434 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.219419 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:52.232718 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.232694 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67h4\" (UniqueName: \"kubernetes.io/projected/79bc020b-b3db-4f47-bc6c-5cdca6b40b8e-kube-api-access-c67h4\") pod \"test-curl-7481-140180322338976\" (UID: \"79bc020b-b3db-4f47-bc6c-5cdca6b40b8e\") " pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:52.241319 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.241285 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:52.333885 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.333848 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67h4\" (UniqueName: \"kubernetes.io/projected/79bc020b-b3db-4f47-bc6c-5cdca6b40b8e-kube-api-access-c67h4\") pod \"test-curl-7481-140180322338976\" (UID: \"79bc020b-b3db-4f47-bc6c-5cdca6b40b8e\") " pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:52.343580 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.343543 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" podUID="133415d6-438a-4ef7-94cd-b7b62da70014" containerName="main" containerID="cri-o://970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821" gracePeriod=45 Jul 21 09:08:52.344947 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.344917 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67h4\" (UniqueName: \"kubernetes.io/projected/79bc020b-b3db-4f47-bc6c-5cdca6b40b8e-kube-api-access-c67h4\") pod \"test-curl-7481-140180322338976\" (UID: \"79bc020b-b3db-4f47-bc6c-5cdca6b40b8e\") " pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:52.358555 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.358531 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08105d94-dd52-473a-9045-56a8a7f681f0" path="/var/lib/kubelet/pods/08105d94-dd52-473a-9045-56a8a7f681f0/volumes" Jul 21 09:08:52.527959 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.527932 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:52.596427 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.596400 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:08:52.637525 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.637495 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-tmp-dir\") pod \"133415d6-438a-4ef7-94cd-b7b62da70014\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " Jul 21 09:08:52.637667 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.637546 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-model-cache\") pod \"133415d6-438a-4ef7-94cd-b7b62da70014\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " Jul 21 09:08:52.637667 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.637572 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-home\") pod \"133415d6-438a-4ef7-94cd-b7b62da70014\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " Jul 21 09:08:52.637667 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.637614 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85xm6\" (UniqueName: \"kubernetes.io/projected/133415d6-438a-4ef7-94cd-b7b62da70014-kube-api-access-85xm6\") pod \"133415d6-438a-4ef7-94cd-b7b62da70014\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " Jul 21 09:08:52.637848 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.637671 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-dshm\") pod \"133415d6-438a-4ef7-94cd-b7b62da70014\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " Jul 21 09:08:52.637848 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.637698 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133415d6-438a-4ef7-94cd-b7b62da70014-tls-certs\") pod \"133415d6-438a-4ef7-94cd-b7b62da70014\" (UID: \"133415d6-438a-4ef7-94cd-b7b62da70014\") " Jul 21 09:08:52.637848 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.637802 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-model-cache" (OuterVolumeSpecName: "model-cache") pod "133415d6-438a-4ef7-94cd-b7b62da70014" (UID: "133415d6-438a-4ef7-94cd-b7b62da70014"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:08:52.637992 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.637808 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "133415d6-438a-4ef7-94cd-b7b62da70014" (UID: "133415d6-438a-4ef7-94cd-b7b62da70014"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:08:52.637992 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.637948 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-home" (OuterVolumeSpecName: "home") pod "133415d6-438a-4ef7-94cd-b7b62da70014" (UID: "133415d6-438a-4ef7-94cd-b7b62da70014"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:08:52.638094 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.638067 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:52.638094 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.638088 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:52.638196 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.638102 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:52.640256 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.640225 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133415d6-438a-4ef7-94cd-b7b62da70014-kube-api-access-85xm6" (OuterVolumeSpecName: "kube-api-access-85xm6") pod "133415d6-438a-4ef7-94cd-b7b62da70014" (UID: "133415d6-438a-4ef7-94cd-b7b62da70014"). InnerVolumeSpecName "kube-api-access-85xm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:08:52.640404 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.640360 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-dshm" (OuterVolumeSpecName: "dshm") pod "133415d6-438a-4ef7-94cd-b7b62da70014" (UID: "133415d6-438a-4ef7-94cd-b7b62da70014"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:08:52.640474 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.640444 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133415d6-438a-4ef7-94cd-b7b62da70014-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "133415d6-438a-4ef7-94cd-b7b62da70014" (UID: "133415d6-438a-4ef7-94cd-b7b62da70014"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:08:52.676510 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.676484 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:52.677586 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:08:52.677560 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79bc020b_b3db_4f47_bc6c_5cdca6b40b8e.slice/crio-3a6de88874d49b0f72be5622a7e507f1af9aa1969124487f8a576aa31ebd7409 WatchSource:0}: Error finding container 3a6de88874d49b0f72be5622a7e507f1af9aa1969124487f8a576aa31ebd7409: Status 404 returned error can't find the container with id 3a6de88874d49b0f72be5622a7e507f1af9aa1969124487f8a576aa31ebd7409 Jul 21 09:08:52.738911 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.738881 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85xm6\" (UniqueName: \"kubernetes.io/projected/133415d6-438a-4ef7-94cd-b7b62da70014-kube-api-access-85xm6\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:52.738911 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.738906 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133415d6-438a-4ef7-94cd-b7b62da70014-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:52.738911 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.738917 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133415d6-438a-4ef7-94cd-b7b62da70014-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:52.858925 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.858885 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322338976" event={"ID":"79bc020b-b3db-4f47-bc6c-5cdca6b40b8e","Type":"ContainerStarted","Data":"3a6de88874d49b0f72be5622a7e507f1af9aa1969124487f8a576aa31ebd7409"} Jul 21 09:08:52.860091 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.860065 2563 generic.go:356] "Generic (PLEG): container finished" podID="133415d6-438a-4ef7-94cd-b7b62da70014" containerID="970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821" exitCode=0 Jul 21 09:08:52.860223 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.860098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" event={"ID":"133415d6-438a-4ef7-94cd-b7b62da70014","Type":"ContainerDied","Data":"970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821"} Jul 21 09:08:52.860223 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.860147 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" event={"ID":"133415d6-438a-4ef7-94cd-b7b62da70014","Type":"ContainerDied","Data":"49f0a66c16898010d927d74c66c0cc7a1e842a7d35c0aff8e57490d523a2008b"} Jul 21 09:08:52.860223 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.860170 2563 scope.go:117] "RemoveContainer" containerID="970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821" Jul 21 09:08:52.860223 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.860109 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9" Jul 21 09:08:52.861615 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.861489 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322337968" Jul 21 09:08:52.869963 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.869950 2563 scope.go:117] "RemoveContainer" containerID="970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821" Jul 21 09:08:52.870208 ip-10-0-137-36 kubenswrapper[2563]: E0721 09:08:52.870191 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821\": container with ID starting with 970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821 not found: ID does not exist" containerID="970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821" Jul 21 09:08:52.870251 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.870218 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821"} err="failed to get container status \"970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821\": rpc error: code = NotFound desc = could not find container \"970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821\": container with ID starting with 970b2658f58c5447a559932f5d7381509f99d7a1c4b7ba2b1aebf1f5f2371821 not found: ID does not exist" Jul 21 09:08:52.870251 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.870234 2563 scope.go:117] "RemoveContainer" containerID="2fb40c462f158ebc628f64e279f852391a88f919a13e8fddff88f5e89457b005" Jul 21 09:08:52.884624 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.884602 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9"] Jul 21 09:08:52.887706 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:52.887685 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-824408/rate-test-model-824408-kserve-5cdbdc84b7-wg9p9"] Jul 21 09:08:53.407440 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.407397 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" podUID="ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" containerName="main" containerID="cri-o://45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511" gracePeriod=45 Jul 21 09:08:53.655631 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.655610 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:08:53.747362 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747272 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-dshm\") pod \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " Jul 21 09:08:53.747362 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747316 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-model-cache\") pod \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " Jul 21 09:08:53.747362 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747364 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tmp-dir\") pod \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " Jul 21 09:08:53.747667 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747437 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tls-certs\") pod \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " Jul 21 09:08:53.747667 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747464 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q57hw\" (UniqueName: \"kubernetes.io/projected/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-kube-api-access-q57hw\") pod \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " Jul 21 09:08:53.747667 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747514 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-home\") pod \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\" (UID: \"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820\") " Jul 21 09:08:53.747667 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747578 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" (UID: "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:08:53.747667 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747600 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" (UID: "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:08:53.747897 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747818 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-home" (OuterVolumeSpecName: "home") pod "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" (UID: "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:08:53.747897 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747854 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-model-cache\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:53.747897 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.747868 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tmp-dir\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:53.749514 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.749489 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-dshm" (OuterVolumeSpecName: "dshm") pod "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" (UID: "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:08:53.749602 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.749542 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" (UID: "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:08:53.749671 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.749654 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-kube-api-access-q57hw" (OuterVolumeSpecName: "kube-api-access-q57hw") pod "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" (UID: "ba4b2aa4-7dba-4d85-8b0b-cb4db2784820"). InnerVolumeSpecName "kube-api-access-q57hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:08:53.848995 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.848959 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-dshm\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:53.848995 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.848987 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-tls-certs\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:53.848995 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.849001 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q57hw\" (UniqueName: \"kubernetes.io/projected/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-kube-api-access-q57hw\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:53.849235 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.849009 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820-home\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:53.868350 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.868317 2563 generic.go:356] "Generic (PLEG): container finished" podID="ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" containerID="45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511" exitCode=0 Jul 21 09:08:53.868537 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.868361 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" Jul 21 09:08:53.868537 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.868429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" event={"ID":"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820","Type":"ContainerDied","Data":"45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511"} Jul 21 09:08:53.868537 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.868484 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc" event={"ID":"ba4b2aa4-7dba-4d85-8b0b-cb4db2784820","Type":"ContainerDied","Data":"e2e838317de91b963a5c8493116a79a8b086428e9ad76751839b63bdf6b17c0c"} Jul 21 09:08:53.868537 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.868512 2563 scope.go:117] "RemoveContainer" containerID="45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511" Jul 21 09:08:53.872488 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.872459 2563 generic.go:356] "Generic (PLEG): container finished" podID="79bc020b-b3db-4f47-bc6c-5cdca6b40b8e" containerID="c10863a45ffb58ea605854306bb79387c5247d0a07e15b3b517d2168097e6294" exitCode=0 Jul 21 09:08:53.872625 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.872498 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322338976" event={"ID":"79bc020b-b3db-4f47-bc6c-5cdca6b40b8e","Type":"ContainerDied","Data":"c10863a45ffb58ea605854306bb79387c5247d0a07e15b3b517d2168097e6294"} Jul 21 09:08:53.878478 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.878460 2563 scope.go:117] "RemoveContainer" containerID="45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511" Jul 21 09:08:53.878718 ip-10-0-137-36 kubenswrapper[2563]: E0721 09:08:53.878701 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511\": container with ID starting with 45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511 not found: ID does not exist" containerID="45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511" Jul 21 09:08:53.878767 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.878728 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511"} err="failed to get container status \"45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511\": rpc error: code = NotFound desc = could not find container \"45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511\": container with ID starting with 45659ac86715c5934287a60117cf8c8834b4a54a8779f741610730c61d48a511 not found: ID does not exist" Jul 21 09:08:53.903213 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.903187 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc"] Jul 21 09:08:53.909916 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:53.909891 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-26961c/rate-test-model-26961c-kserve-766594c859-v49sc"] Jul 21 09:08:54.363987 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:54.363956 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="133415d6-438a-4ef7-94cd-b7b62da70014" path="/var/lib/kubelet/pods/133415d6-438a-4ef7-94cd-b7b62da70014/volumes" Jul 21 09:08:54.364352 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:54.364339 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" path="/var/lib/kubelet/pods/ba4b2aa4-7dba-4d85-8b0b-cb4db2784820/volumes" Jul 21 09:08:55.024871 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.024847 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:55.051320 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.051295 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-140180322338976_79bc020b-b3db-4f47-bc6c-5cdca6b40b8e/test-curl-7481-140180322338976/0.log" Jul 21 09:08:55.058674 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.058650 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c67h4\" (UniqueName: \"kubernetes.io/projected/79bc020b-b3db-4f47-bc6c-5cdca6b40b8e-kube-api-access-c67h4\") pod \"79bc020b-b3db-4f47-bc6c-5cdca6b40b8e\" (UID: \"79bc020b-b3db-4f47-bc6c-5cdca6b40b8e\") " Jul 21 09:08:55.060904 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.060882 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bc020b-b3db-4f47-bc6c-5cdca6b40b8e-kube-api-access-c67h4" (OuterVolumeSpecName: "kube-api-access-c67h4") pod "79bc020b-b3db-4f47-bc6c-5cdca6b40b8e" (UID: "79bc020b-b3db-4f47-bc6c-5cdca6b40b8e"). InnerVolumeSpecName "kube-api-access-c67h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:08:55.080309 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.080287 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:55.084397 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.084366 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:55.159747 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.159695 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c67h4\" (UniqueName: \"kubernetes.io/projected/79bc020b-b3db-4f47-bc6c-5cdca6b40b8e-kube-api-access-c67h4\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:55.217820 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.217786 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:55.218122 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.218110 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133415d6-438a-4ef7-94cd-b7b62da70014" containerName="main" Jul 21 09:08:55.218159 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.218124 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="133415d6-438a-4ef7-94cd-b7b62da70014" containerName="main" Jul 21 09:08:55.218159 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.218135 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" containerName="main" Jul 21 09:08:55.218159 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.218142 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" containerName="main" Jul 21 09:08:55.218159 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.218150 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bc020b-b3db-4f47-bc6c-5cdca6b40b8e" containerName="test-curl-7481-140180322338976" Jul 21 09:08:55.218159 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.218158 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bc020b-b3db-4f47-bc6c-5cdca6b40b8e" containerName="test-curl-7481-140180322338976" Jul 21 09:08:55.218356 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.218220 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba4b2aa4-7dba-4d85-8b0b-cb4db2784820" containerName="main" Jul 21 09:08:55.218356 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.218226 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="133415d6-438a-4ef7-94cd-b7b62da70014" containerName="main" Jul 21 09:08:55.218356 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.218232 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bc020b-b3db-4f47-bc6c-5cdca6b40b8e" containerName="test-curl-7481-140180322338976" Jul 21 09:08:55.221104 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.221088 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:55.229997 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.229970 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:55.260992 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.260967 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9hz2\" (UniqueName: \"kubernetes.io/projected/4bfe2f08-030e-42f0-a8fb-63decbea7f14-kube-api-access-q9hz2\") pod \"test-curl-7481-140180322338976\" (UID: \"4bfe2f08-030e-42f0-a8fb-63decbea7f14\") " pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:55.361505 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.361458 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9hz2\" (UniqueName: \"kubernetes.io/projected/4bfe2f08-030e-42f0-a8fb-63decbea7f14-kube-api-access-q9hz2\") pod \"test-curl-7481-140180322338976\" (UID: \"4bfe2f08-030e-42f0-a8fb-63decbea7f14\") " pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:55.388997 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.388974 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9hz2\" (UniqueName: \"kubernetes.io/projected/4bfe2f08-030e-42f0-a8fb-63decbea7f14-kube-api-access-q9hz2\") pod \"test-curl-7481-140180322338976\" (UID: \"4bfe2f08-030e-42f0-a8fb-63decbea7f14\") " pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:55.531362 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.531268 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:55.667334 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.667310 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:55.668511 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:08:55.668490 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bfe2f08_030e_42f0_a8fb_63decbea7f14.slice/crio-b0545c7b2b941ef5c982e1b31b534a892eb03a92a597daae006397ada9baae4e WatchSource:0}: Error finding container b0545c7b2b941ef5c982e1b31b534a892eb03a92a597daae006397ada9baae4e: Status 404 returned error can't find the container with id b0545c7b2b941ef5c982e1b31b534a892eb03a92a597daae006397ada9baae4e Jul 21 09:08:55.885373 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.885285 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:55.885373 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.885283 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6de88874d49b0f72be5622a7e507f1af9aa1969124487f8a576aa31ebd7409" Jul 21 09:08:55.886557 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.886521 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322338976" event={"ID":"4bfe2f08-030e-42f0-a8fb-63decbea7f14","Type":"ContainerStarted","Data":"b0545c7b2b941ef5c982e1b31b534a892eb03a92a597daae006397ada9baae4e"} Jul 21 09:08:55.901019 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:55.900995 2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7481-140180322338976" oldPodUID="79bc020b-b3db-4f47-bc6c-5cdca6b40b8e" podUID="4bfe2f08-030e-42f0-a8fb-63decbea7f14" Jul 21 09:08:56.357442 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:56.357412 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79bc020b-b3db-4f47-bc6c-5cdca6b40b8e" path="/var/lib/kubelet/pods/79bc020b-b3db-4f47-bc6c-5cdca6b40b8e/volumes" Jul 21 09:08:56.893885 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:56.893851 2563 generic.go:356] "Generic (PLEG): container finished" podID="4bfe2f08-030e-42f0-a8fb-63decbea7f14" containerID="e23e8c49f7b64b76cbbc481252bc0cd5c3f073301975d231cc07c8fc5968850a" exitCode=0 Jul 21 09:08:56.894088 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:56.893935 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-140180322338976" event={"ID":"4bfe2f08-030e-42f0-a8fb-63decbea7f14","Type":"ContainerDied","Data":"e23e8c49f7b64b76cbbc481252bc0cd5c3f073301975d231cc07c8fc5968850a"} Jul 21 09:08:58.039735 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.039709 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:58.080301 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.080281 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-140180322338976_4bfe2f08-030e-42f0-a8fb-63decbea7f14/test-curl-7481-140180322338976/0.log" Jul 21 09:08:58.082548 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.082529 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9hz2\" (UniqueName: \"kubernetes.io/projected/4bfe2f08-030e-42f0-a8fb-63decbea7f14-kube-api-access-q9hz2\") pod \"4bfe2f08-030e-42f0-a8fb-63decbea7f14\" (UID: \"4bfe2f08-030e-42f0-a8fb-63decbea7f14\") " Jul 21 09:08:58.084716 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.084694 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfe2f08-030e-42f0-a8fb-63decbea7f14-kube-api-access-q9hz2" (OuterVolumeSpecName: "kube-api-access-q9hz2") pod "4bfe2f08-030e-42f0-a8fb-63decbea7f14" (UID: "4bfe2f08-030e-42f0-a8fb-63decbea7f14"). InnerVolumeSpecName "kube-api-access-q9hz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:08:58.115851 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.115827 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:58.126079 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.126052 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-140180322338976"] Jul 21 09:08:58.183961 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.183885 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9hz2\" (UniqueName: \"kubernetes.io/projected/4bfe2f08-030e-42f0-a8fb-63decbea7f14-kube-api-access-q9hz2\") on node \"ip-10-0-137-36.ec2.internal\" DevicePath \"\"" Jul 21 09:08:58.358693 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.358661 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bfe2f08-030e-42f0-a8fb-63decbea7f14" path="/var/lib/kubelet/pods/4bfe2f08-030e-42f0-a8fb-63decbea7f14/volumes" Jul 21 09:08:58.905508 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.905475 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-140180322338976" Jul 21 09:08:58.905508 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:58.905508 2563 scope.go:117] "RemoveContainer" containerID="e23e8c49f7b64b76cbbc481252bc0cd5c3f073301975d231cc07c8fc5968850a" Jul 21 09:08:59.261525 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:59.261429 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:08:59.363901 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:08:59.363871 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:08.874693 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:08.874638 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:11.747111 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:11.747073 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:09:13.444495 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:13.444451 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:13.850813 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:13.850710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:20.253970 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:20.253926 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:20.326783 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:20.326748 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:20.561984 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:20.561895 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:20.808214 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:20.808176 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:21.458341 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:21.458309 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:21.563706 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:21.563678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:31.475733 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:31.475694 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:34.440062 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:34.440031 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rm5h8"] Jul 21 09:09:36.036515 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:36.036481 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:36.460562 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:36.460531 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:42.227330 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:42.227247 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:42.350582 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:42.350543 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:42.726940 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:42.726913 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:42.804965 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:42.804934 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:43.042086 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:43.042001 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:43.269775 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:43.269733 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:09:50.150488 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:50.150460 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787895fdf5-brfgk_98373695-9159-4926-9b85-c37aaeb4bc3c/copy-manifests/0.log" Jul 21 09:09:50.169508 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:50.169477 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787895fdf5-brfgk_98373695-9159-4926-9b85-c37aaeb4bc3c/manager/0.log" Jul 21 09:09:51.969664 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:51.969633 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp_6f46db13-b76f-412d-bc4b-1d4477609776/util/0.log" Jul 21 09:09:51.978396 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:51.978365 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp_6f46db13-b76f-412d-bc4b-1d4477609776/pull/0.log" Jul 21 09:09:51.988452 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:51.988433 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp_6f46db13-b76f-412d-bc4b-1d4477609776/extract/0.log" Jul 21 09:09:52.103591 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.103558 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n_cba30917-00d2-477b-8a8d-9428d179dc34/extract/0.log" Jul 21 09:09:52.110827 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.110805 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n_cba30917-00d2-477b-8a8d-9428d179dc34/util/0.log" Jul 21 09:09:52.120415 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.120374 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n_cba30917-00d2-477b-8a8d-9428d179dc34/pull/0.log" Jul 21 09:09:52.232840 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.232764 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c_b85e6820-7dbb-4da0-8fbd-809f3b06bc61/util/0.log" Jul 21 09:09:52.240452 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.240423 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c_b85e6820-7dbb-4da0-8fbd-809f3b06bc61/pull/0.log" Jul 21 09:09:52.250246 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.250224 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c_b85e6820-7dbb-4da0-8fbd-809f3b06bc61/extract/0.log" Jul 21 09:09:52.368324 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.368294 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn_5d16c9bc-9cee-4761-b59d-133d91538fbc/util/0.log" Jul 21 09:09:52.375138 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.375115 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn_5d16c9bc-9cee-4761-b59d-133d91538fbc/pull/0.log" Jul 21 09:09:52.382711 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.382689 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn_5d16c9bc-9cee-4761-b59d-133d91538fbc/extract/0.log" Jul 21 09:09:52.736991 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.736956 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-lndf4_83f55247-7642-48ab-abf8-5c829f324abf/manager/0.log" Jul 21 09:09:52.865813 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:52.865788 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-mhpf6_7a90306f-e1ff-48fc-8a80-edd9386fb7ce/kuadrant-console-plugin/0.log" Jul 21 09:09:53.204805 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:53.204779 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-rm5h8_3e681ce3-b040-42ba-825b-18ed78d49fad/limitador/0.log" Jul 21 09:09:53.911843 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:53.911814 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d57bc4cd9-bw7rq_3050721d-8325-4e44-a7f5-376af01c2765/kube-auth-proxy/0.log" Jul 21 09:09:54.483466 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:54.483436 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8566ccfd7b-fdt46_bae2b6a9-8844-42aa-9c78-1d931caed9d4/router/0.log" Jul 21 09:09:55.211197 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:55.211165 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjmrt7_ca6a1159-57d8-46cc-b1d1-61cdb48b23e8/main/0.log" Jul 21 09:09:55.451014 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:09:55.450986 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-8rfcb_3bd2bef4-3248-43ea-bf12-70ed6ad88965/main/0.log" Jul 21 09:10:01.220297 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.220253 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xts55/must-gather-c5hsz"] Jul 21 09:10:01.220805 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.220749 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bfe2f08-030e-42f0-a8fb-63decbea7f14" containerName="test-curl-7481-140180322338976" Jul 21 09:10:01.220805 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.220768 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bfe2f08-030e-42f0-a8fb-63decbea7f14" containerName="test-curl-7481-140180322338976" Jul 21 09:10:01.220925 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.220887 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bfe2f08-030e-42f0-a8fb-63decbea7f14" containerName="test-curl-7481-140180322338976" Jul 21 09:10:01.223870 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.223841 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xts55/must-gather-c5hsz" Jul 21 09:10:01.228399 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.228347 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xts55\"/\"default-dockercfg-zhkmm\"" Jul 21 09:10:01.228540 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.228526 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xts55\"/\"openshift-service-ca.crt\"" Jul 21 09:10:01.229431 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.229416 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xts55\"/\"kube-root-ca.crt\"" Jul 21 09:10:01.234761 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.234742 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xts55/must-gather-c5hsz"] Jul 21 09:10:01.240760 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.240722 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:10:01.326286 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.326252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/928e6bf9-a9cb-4404-92b0-251a87fc17f0-must-gather-output\") pod \"must-gather-c5hsz\" (UID: \"928e6bf9-a9cb-4404-92b0-251a87fc17f0\") " pod="openshift-must-gather-xts55/must-gather-c5hsz" Jul 21 09:10:01.326489 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.326298 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbrl\" (UniqueName: \"kubernetes.io/projected/928e6bf9-a9cb-4404-92b0-251a87fc17f0-kube-api-access-rfbrl\") pod \"must-gather-c5hsz\" (UID: \"928e6bf9-a9cb-4404-92b0-251a87fc17f0\") " pod="openshift-must-gather-xts55/must-gather-c5hsz" Jul 21 09:10:01.339217 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.339187 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:10:01.427296 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.427265 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbrl\" (UniqueName: \"kubernetes.io/projected/928e6bf9-a9cb-4404-92b0-251a87fc17f0-kube-api-access-rfbrl\") pod \"must-gather-c5hsz\" (UID: \"928e6bf9-a9cb-4404-92b0-251a87fc17f0\") " pod="openshift-must-gather-xts55/must-gather-c5hsz" Jul 21 09:10:01.427498 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.427349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/928e6bf9-a9cb-4404-92b0-251a87fc17f0-must-gather-output\") pod \"must-gather-c5hsz\" (UID: \"928e6bf9-a9cb-4404-92b0-251a87fc17f0\") " pod="openshift-must-gather-xts55/must-gather-c5hsz" Jul 21 09:10:01.427670 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.427652 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/928e6bf9-a9cb-4404-92b0-251a87fc17f0-must-gather-output\") pod \"must-gather-c5hsz\" (UID: \"928e6bf9-a9cb-4404-92b0-251a87fc17f0\") " pod="openshift-must-gather-xts55/must-gather-c5hsz" Jul 21 09:10:01.436342 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.436320 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbrl\" (UniqueName: \"kubernetes.io/projected/928e6bf9-a9cb-4404-92b0-251a87fc17f0-kube-api-access-rfbrl\") pod \"must-gather-c5hsz\" (UID: \"928e6bf9-a9cb-4404-92b0-251a87fc17f0\") " pod="openshift-must-gather-xts55/must-gather-c5hsz" Jul 21 09:10:01.533864 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.533777 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xts55/must-gather-c5hsz" Jul 21 09:10:01.672084 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:01.672056 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xts55/must-gather-c5hsz"] Jul 21 09:10:01.674146 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:10:01.674117 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod928e6bf9_a9cb_4404_92b0_251a87fc17f0.slice/crio-b8a0a50f2a2870f4327605f130f9066d9c0ee48d3f3ffaa179dd3247f8d10e8a WatchSource:0}: Error finding container b8a0a50f2a2870f4327605f130f9066d9c0ee48d3f3ffaa179dd3247f8d10e8a: Status 404 returned error can't find the container with id b8a0a50f2a2870f4327605f130f9066d9c0ee48d3f3ffaa179dd3247f8d10e8a Jul 21 09:10:02.233576 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:02.233536 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xts55/must-gather-c5hsz" event={"ID":"928e6bf9-a9cb-4404-92b0-251a87fc17f0","Type":"ContainerStarted","Data":"b8a0a50f2a2870f4327605f130f9066d9c0ee48d3f3ffaa179dd3247f8d10e8a"} Jul 21 09:10:03.248976 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:03.248938 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xts55/must-gather-c5hsz" event={"ID":"928e6bf9-a9cb-4404-92b0-251a87fc17f0","Type":"ContainerStarted","Data":"87954fde17c4e69be029b3b347902031b58df992cb1b91fc9f19ec85dcc1ad3c"} Jul 21 09:10:03.249434 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:03.248987 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xts55/must-gather-c5hsz" event={"ID":"928e6bf9-a9cb-4404-92b0-251a87fc17f0","Type":"ContainerStarted","Data":"7e556b291bbef110d96e2f37c3d2725a1bb1294b27d2471daf41c3976dc0e877"} Jul 21 09:10:03.275996 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:03.275936 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xts55/must-gather-c5hsz" podStartSLOduration=1.226172328 podStartE2EDuration="2.275918874s" totalImagesPullingTime="1.049746546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:10:03.275711491 +0000 UTC m=+3075.354749823" watchObservedRunningTime="2026-07-21 09:10:03.275918874 +0000 UTC m=+3075.354957205" Jul 21 09:10:04.423110 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:04.423065 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fkhp9_ae3fb7e3-9363-4e30-b5c4-362303bd14bb/global-pull-secret-syncer/0.log" Jul 21 09:10:04.521315 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:04.521286 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" Jul 21 09:10:04.636371 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:04.636329 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-36.ec2.internal_562d6f5337e483c37d48c2d1b5b30966/haproxy/0.log" Jul 21 09:10:07.527710 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:07.527675 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m5l9v_39eb937b-c6b2-47bf-97ab-7ccd168023ec/konnectivity-agent/0.log" limit=1024 Jul 21 09:10:09.312120 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.312088 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp_6f46db13-b76f-412d-bc4b-1d4477609776/extract/0.log" Jul 21 09:10:09.338147 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.338108 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp_6f46db13-b76f-412d-bc4b-1d4477609776/util/0.log" Jul 21 09:10:09.365346 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.365312 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pn9zp_6f46db13-b76f-412d-bc4b-1d4477609776/pull/0.log" Jul 21 09:10:09.415983 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.415937 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n_cba30917-00d2-477b-8a8d-9428d179dc34/extract/0.log" Jul 21 09:10:09.447339 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.447297 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n_cba30917-00d2-477b-8a8d-9428d179dc34/util/0.log" Jul 21 09:10:09.485476 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.485443 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s464n_cba30917-00d2-477b-8a8d-9428d179dc34/pull/0.log" Jul 21 09:10:09.536168 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.536138 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c_b85e6820-7dbb-4da0-8fbd-809f3b06bc61/extract/0.log" Jul 21 09:10:09.572184 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.572104 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c_b85e6820-7dbb-4da0-8fbd-809f3b06bc61/util/0.log" Jul 21 09:10:09.605408 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.605362 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mjr5c_b85e6820-7dbb-4da0-8fbd-809f3b06bc61/pull/0.log" Jul 21 09:10:09.643333 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.643298 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn_5d16c9bc-9cee-4761-b59d-133d91538fbc/extract/0.log" Jul 21 09:10:09.670674 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.670641 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn_5d16c9bc-9cee-4761-b59d-133d91538fbc/util/0.log" Jul 21 09:10:09.696567 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:09.696534 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14wmqn_5d16c9bc-9cee-4761-b59d-133d91538fbc/pull/0.log" Jul 21 09:10:10.131029 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:10.130996 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-lndf4_83f55247-7642-48ab-abf8-5c829f324abf/manager/0.log" Jul 21 09:10:10.164632 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:10.164471 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-mhpf6_7a90306f-e1ff-48fc-8a80-edd9386fb7ce/kuadrant-console-plugin/0.log" Jul 21 09:10:10.417062 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:10.417030 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-rm5h8_3e681ce3-b040-42ba-825b-18ed78d49fad/limitador/0.log" Jul 21 09:10:12.606785 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:12.606751 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5dd65855c4-dfhcn_1fb3f2f4-48d4-4918-9aef-f4312bcd9f89/metrics-server/0.log" Jul 21 09:10:13.030013 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:13.029988 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zndk7_fc259ca0-b711-40b9-9c06-4ec62ca949f9/node-exporter/0.log" Jul 21 09:10:13.123025 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:13.122993 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zndk7_fc259ca0-b711-40b9-9c06-4ec62ca949f9/kube-rbac-proxy/0.log" Jul 21 09:10:13.159552 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:13.159525 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zndk7_fc259ca0-b711-40b9-9c06-4ec62ca949f9/init-textfile/0.log" Jul 21 09:10:13.688533 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:13.688502 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-td5t4_e86fc848-86bc-45a2-b378-1c543ff46a97/prometheus-operator/0.log" Jul 21 09:10:13.722898 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:13.722865 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-td5t4_e86fc848-86bc-45a2-b378-1c543ff46a97/kube-rbac-proxy/0.log" Jul 21 09:10:13.764956 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:13.764928 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-k726q_c7dad7be-0b60-4a0e-9139-cef84cc4f391/prometheus-operator-admission-webhook/0.log" Jul 21 09:10:14.479923 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.479884 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj"] Jul 21 09:10:14.483782 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.483750 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.495523 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.495494 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj"] Jul 21 09:10:14.566996 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.566956 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-proc\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.567202 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.567015 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-lib-modules\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.567202 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.567082 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-sys\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.567202 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.567126 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmsjw\" (UniqueName: \"kubernetes.io/projected/0e131719-3379-4295-8ef1-3f7262c7ce70-kube-api-access-tmsjw\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.567202 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.567153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-podres\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.667651 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.667615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-sys\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.667950 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.667929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmsjw\" (UniqueName: \"kubernetes.io/projected/0e131719-3379-4295-8ef1-3f7262c7ce70-kube-api-access-tmsjw\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.668095 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.668079 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-podres\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.668472 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.668234 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-podres\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.668472 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.668253 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-proc\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.668472 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.667761 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-sys\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.668472 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.668317 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-proc\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.668472 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.668319 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-lib-modules\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.668472 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.668442 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e131719-3379-4295-8ef1-3f7262c7ce70-lib-modules\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.680777 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.680750 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmsjw\" (UniqueName: \"kubernetes.io/projected/0e131719-3379-4295-8ef1-3f7262c7ce70-kube-api-access-tmsjw\") pod \"perf-node-gather-daemonset-6q8tj\" (UID: \"0e131719-3379-4295-8ef1-3f7262c7ce70\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.798044 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.797950 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:14.998781 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:14.998255 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj"] Jul 21 09:10:15.003857 ip-10-0-137-36 kubenswrapper[2563]: W0721 09:10:15.003819 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0e131719_3379_4295_8ef1_3f7262c7ce70.slice/crio-044396707aa77ec654e8c7bd6a39b4f7a7242c2e61823ccde9ca6cc783ee1cbb WatchSource:0}: Error finding container 044396707aa77ec654e8c7bd6a39b4f7a7242c2e61823ccde9ca6cc783ee1cbb: Status 404 returned error can't find the container with id 044396707aa77ec654e8c7bd6a39b4f7a7242c2e61823ccde9ca6cc783ee1cbb Jul 21 09:10:15.347911 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:15.343993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" event={"ID":"0e131719-3379-4295-8ef1-3f7262c7ce70","Type":"ContainerStarted","Data":"fedd8608a804f8bbcf8e1bdb5fcfd7b83189b79ac395fa1282f72cca63bdc959"} Jul 21 09:10:15.347911 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:15.344078 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" event={"ID":"0e131719-3379-4295-8ef1-3f7262c7ce70","Type":"ContainerStarted","Data":"044396707aa77ec654e8c7bd6a39b4f7a7242c2e61823ccde9ca6cc783ee1cbb"} Jul 21 09:10:15.347911 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:15.344608 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:15.364491 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:15.364411 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" podStartSLOduration=1.364376545 podStartE2EDuration="1.364376545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 09:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:10:15.363633611 +0000 UTC m=+3087.442671951" watchObservedRunningTime="2026-07-21 09:10:15.364376545 +0000 UTC m=+3087.443414877" Jul 21 09:10:15.378860 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:15.378830 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-kwwxv_f8bcd643-8b1e-4672-a68b-759c36bfad9d/networking-console-plugin/0.log" Jul 21 09:10:16.401477 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:16.401442 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d88bb864f-clkqf_0d5e055d-5c1b-4a00-9530-b4c6713426c3/console/0.log" Jul 21 09:10:18.278165 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:18.278130 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xws6r_0f822e8b-5f1a-4335-8b98-928c1fb36ced/dns/0.log" Jul 21 09:10:18.336950 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:18.336920 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xws6r_0f822e8b-5f1a-4335-8b98-928c1fb36ced/kube-rbac-proxy/0.log" Jul 21 09:10:18.452334 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:18.452307 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-llppt_a59731f0-2456-4447-a672-69a180edc896/dns-node-resolver/0.log" Jul 21 09:10:19.059245 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:19.059208 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5m66_c044a2bf-898d-4e0c-b624-01d347f5d5f9/node-ca/0.log" Jul 21 09:10:20.263594 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:20.263567 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d57bc4cd9-bw7rq_3050721d-8325-4e44-a7f5-376af01c2765/kube-auth-proxy/0.log" Jul 21 09:10:20.455041 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:20.455008 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8566ccfd7b-fdt46_bae2b6a9-8844-42aa-9c78-1d931caed9d4/router/0.log" Jul 21 09:10:21.291960 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:21.291922 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8kwc_041c75cc-9740-48f5-8f2b-12043420de1e/serve-healthcheck-canary/0.log" Jul 21 09:10:21.367407 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:21.367363 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-6q8tj" Jul 21 09:10:21.798450 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:21.798420 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-nxz4z_115156cb-6b04-420e-a460-b61a69906ef2/insights-operator/0.log" Jul 21 09:10:21.798979 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:21.798959 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-nxz4z_115156cb-6b04-420e-a460-b61a69906ef2/insights-operator/1.log" Jul 21 09:10:21.898093 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:21.898068 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgk7x_65710829-45b3-467d-ba2c-5b3e02fde342/kube-rbac-proxy/0.log" Jul 21 09:10:21.920632 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:21.920589 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgk7x_65710829-45b3-467d-ba2c-5b3e02fde342/exporter/0.log" Jul 21 09:10:21.946999 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:21.946973 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgk7x_65710829-45b3-467d-ba2c-5b3e02fde342/extractor/0.log" Jul 21 09:10:24.848050 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:24.848018 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d57bc4cd9-bw7rq_3050721d-8325-4e44-a7f5-376af01c2765/kube-auth-proxy/0.log" Jul 21 09:10:25.020129 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:25.020089 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8566ccfd7b-fdt46_bae2b6a9-8844-42aa-9c78-1d931caed9d4/router/0.log" Jul 21 09:10:25.513057 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:25.513020 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787895fdf5-brfgk_98373695-9159-4926-9b85-c37aaeb4bc3c/manager/0.log" Jul 21 09:10:25.532569 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:25.532537 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787895fdf5-brfgk_98373695-9159-4926-9b85-c37aaeb4bc3c/copy-manifests/0.log" Jul 21 09:10:26.434537 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:26.434511 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d57bc4cd9-bw7rq_3050721d-8325-4e44-a7f5-376af01c2765/kube-auth-proxy/0.log" Jul 21 09:10:26.618039 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:26.617970 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8566ccfd7b-fdt46_bae2b6a9-8844-42aa-9c78-1d931caed9d4/router/0.log" Jul 21 09:10:27.060441 ip-10-0-137-36 kubenswrapper[2563]: I0721 09:10:27.060407 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7686776559-xvtl4_612a5640-1975-4c4a-b9e5-9722e4c2d3c3/manager/0.log"